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1.0 INTRODUCTION 


1.1 OBJECTIVES AND PROPOSAL ORGANI ZATION 


Many military decision support systems (DSS's) have been developed 
during the past decade but few Studies have been undertaken to determine 
their worth. There has been virtually no work aimed at developing standardized 
measures of effectiveness and standardized types of experiments for comparing 
decisions made with and without DSS's. This fs a proposal for a two-year 7 
program to develop tools for evaluating DSS's and to identify the types of 


military decisions that can benefit most from a DSS. 


The specific objectives of the two-year program are: 


1. Develop theoretical foundations for the quantitative evaluation 
of DSS's including the specification of verifiable hypotheses and the develop- 


ment of objective performance measures. 


2. Develop generalizable experimental procedures for establishing the 
quantitative benefit of DSS's to decision makers in terms of (a) expected 
performance improvement over unaided decisions, (b) user time required to 
make good decisions, and (c) training and experience levels required to 


make good decisions using DSS's. 


3. Develop a taxonomy of types of military decisions that can bene- 
fit most from application of DSS's and criteria for incorporation of DSS's 


into current military operational and training programs and systems. 


The remainder of Section | of the proposal presents an overview of 
the program. Section 2 describes the technical approach in detail. Program 
schedules, a milestone chart, anda list of deliverables are presented in 
Section 3. Section 4 presents an overview of the management and qualifications 
of ISC to perform the prescribed tasks. References appear in Section 5. The 
Appendices include two papers describing research pertaining to the decision 


Support system philosophy that is relevant to the proposed work. 


132 BACKGROUND 


The problem of subjecting decision-aiding techniques to operational 
tests, especially when value judgments are employed, has haunted the 
~Decision-Analysis community since its formation and no satisfactory solution 
has yet been agreed upon. (Some of the basic difficulties in administering 
such tests are discussed in Appendices 1} and 2 which also contain conceptual 
solutions to this problem via a pragmatic interpretation of value judgments.) 
The decision support systems that are the object and focus of the proposed 
program are computerized aids that help a decision maker with two major 
functions: (1) problem structuring and (2) alternative evaluation and selec- 
tion. Systems that only help organize, compute, and display data are | 
"informational aids'' in the sense that they provide the pecetediy background 
for a deciston, but do not address directly the decision process itself. 
Further, the types of problems that decision support systems address are 
usually perceived by the user to consist of a comp lex network of relationships 
with variable and uncertain strengths. The final product is a recommendation 
for the selection of a particular course of action or a strategy plan, rather 
than a numerical output. Decision support systems are usually built to 
satisfy fuzzy or inexact criteria, which may even change during the decision 


process. 


One of the primary features of the decision Support system approach 
is the requirement for the decisionmaker to decompose his decision problem 


into component parts including consideration of: 


@e Relevant problem details 
e Important related issues 


e Possible side effects 


This is called "problem structuring." It is usually assumed by decision 
analysts that the problem structuring process of forcing the decisionmaker 

to consider one aspect of the problem at a time and articulate his judgments 
regarding each aspect, leads to better overall decisions when these judgments © 


are aggregated by mechanical means. In a DSS, the one-at-a-time judgments made 


by the user are usually in the form of utilities and probabilities. The 
computer can quickly combine the local numerical judgments into a global 


recommendatton. 


One of the major areas of research and experimentation is based on 
this conjecture: better decisions are made from a series of local judgments 
that are mathematically aggregated than those obtained by letting the decision- 
maker exercise global judgments. Leal (Reference 1) has shown that in cases” 
of team decisionmaking, groups not only formulate a rich set of decision alter- 
natives when using a structuring aid, but also tend to focus their attention 
on the more critical issues. Armstrong (Reference 2) has shown that estimates 
of factual quantities produced by decomposition were more. valid in most 
cases than those obtained by direct assessments. However, Dalkey (Reference 3) 
has shown in experiments that, in certain cases, decisionmakers actually per- 
form better when making global judgments than when considering problem details. 
Thus, there is no clear criterion for determining when a problem structuring 
function of a decision sypport system will help or actually hinder a decision- 


maker. 


1.3 OVERVIEW OF TECHNICAL APPROACH 


The first step toward developing tools for evaluating the merit and 
beneficial application of DSS's is to construct the theoretical foundations 


for objective evaluation. This Involves establishing: 
1. Verifiable hypotheses of decision system merit. 


2. Quantitative performance measures for comprehensive comparative 


evaluation of DSS and unaided decisionmaking. 


The hypotheses link the problem structuring techniques with their 
effects on the performance of the decisionmaker. Once established, they form 
the basis for the experimental plan and the specification of criteria for 
deciding when DSS's are useful. The quantitative performance measures enable — 
controlled experimentation to be directed toward the satisfaction of DSS 
evaluation objectives. Examples of numerical measures for judging the effec- | 


tiveness of DSS's and unaided systems include: 
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Decision Quality 
Judgment Consistency 


Judgment Validity. 


Decision Time 


These measures and others will form the basis for the methodology and guide- 


lines for DSS evaluation and effectiveness criteria. 


A simulated decision environment is necessary in order to generate 
experimental values of performance measures such as those mentioned above. 
One of the most important elements of the environment is the existence of a 
"correct" answer to a specific decision problem. In the absence of an agreed 
"correct'' answer, DSS evaluations have so far been limited to experts’ assess- 
ments of decision quality. Additional background and theoretical basis for 
the correct answer requirement are given in Subsection 2.5. 

The major work in developing a DSS test bed will be construction of 
a ''scenario generator.'' The scenario generator is computer software that 
sets up the experimentation environment in order to get the most information 
about the DSS performance measures of interest. By varying generator para- 
meters, different environments can be simulated to obtain data on specified 


measures. 


The scenario generator will enable a series of controlled experiments 
to be performed that will determine the relative merits of several decision 
systems as well as determine the kinds of applications where each type of 
decision system can be most beneficially used. Decision performance wil] | 
be monitored and measured in situations involving decisionmaking with support 
and without support. Performance measures will be made all along the subject's 
learning curve from first acquaintance with the simulation environment to 
experienced usage gained from repeated trials. A candidate scenario generator 
based on ongoing work at ISC is described in Section 2.2. : 

The final results from experimental data analysis will, in addition 
to providing a comprehensive picture of the merit of decision support systems, 


form a basis for a general methodology for the evaluation and application of 


she 


future decision aids and support systems. This methodology can be applied to 
existing military operational aids as well] as to training systems to allow 


objective judgments of their benefit and effectiveness. 


2.0 TECHNICAL APPROACH 


This section presents a description of the proposed ‘technical approach 
to evaluating the effectiveness of military decision support systems. The 
major elements of the approach are (1) theoretical foundations, (2) scenario 
generator, (3) evaluation measures and data analysis, and (4) evaluation 


experiments. 


2.1 THEORETICAL FOUNDATIONS 


Before meaningful experimentation and methodology development can take 
place, a firm foundation for decision support system evaluation must be estab- 
lished. This foundation will consist of several basic verifiable hypotheses 
which are related to the benefits and applications of decision support systems. 
The hypotheses will serve to direct the construction of the simulated evalua- 


° 


tion environment. 


As an illustration of a typical verifiable hypothesis, one of the objec- 
tives of the program will be to test the hypothesis that "the most benefit from 
decision support systems is to be gained in the middle of the learning curve." 
Figure 1 shows two superimposed curves. The dotted line its the expected learn- 
ing curve of a typical subject during the course of an experiment with a given 
task. Near the beginning, his progress is slow due to his unfamiliarity 
with the simulation environment. As his experience increases, he is able to 
progress at a faster rate because he has acquired means of classifying and 
interpreting new experiences. Finally, small improvements can still be made 


by ''fine-tuning'’ the parameters of his developed conceptual framework. 


The solid line in the figure represents what we believe to be the 
potential help that a decision support system can give the subject during 
his progress on a large class of decision tasks. Near the beginning of the 
interaction, the subject will be using most of his mental powers to become 
aware of the possible decision alternatives available to him and their impact 
on the environment and on the subject's perceived utility. A deciston support 


structuring aid cannot be of much help since he has not yet formed a mental | 
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learning curve 


potential aid - 
“a of decision 
| support systems 


(structuring aid) 


task task "expertise! 
Initiation familiarity 


(local judgments (local judgments cipher ea 
unreliable) reliable, global ai 
judgments 


unreliable) 


(limited (Model driven (operational 
knowledge) knowledge knowledge 
"what will happen if...) "what to do to achieve X'') 


Figure 1. Potential Aid of Decision Support Systems. 


picture of relevant relattonships which govern the environment. Consequently, 
the local judgments he would be asked to issue are bound to be unreliable, 
resulting In error accumulation through the aggregation process and an inferior 


overall performance. 


The peak of improvement with a decision support system should occur 
after enough task familiarity has been gained to produce reliable models of 
segments of the task domain. However, the subject does not yet have the 
wealth of experience which will allow him to make reliable global or opera- 
tional judgments. Thus, a structuring aid that provides computational aggrega- 


tion of local judgments will be of great benefit. 


Near the end of the learning curve, the subject typically attains a 

level of expertise!’ in the area and, drawing from his experience, is able 

to make reliable global judgments without the need to decompose the decision 
problem. We may picture the final phase of the skill acquisition process as 

a transformation (e.g., recompilation) of the user's knowledge representation 
(e.g., data structure). In intermediate phases his knowledge can be regarded 
8s fragmented chunks of factual, declarative relationships, for example, 

"If action X, then usually effect Y.'' The format in the final phase is goal- 
driven and procedural. For example, ''To achieve effect Y, do X' or, if Z is 
Observable, do X/' According to this model of format transformation, the benefit 
of a structuring aid should decrease at the final phases of skill acquisition. 
It may be the case that in such circumstances, an aid will actually hinder 

the decision maker in achieving a high level of performance. This hypothesis 
remains to be tested. It’ is through these and other verifiable hypotheses, 
that we hope to identify areas where decision support systems could be utilized 


effectively. 


2.2 SCENARIO GENERATOR 


An ideal environment for exercising the experiments proposed is a_ 
“niljtary simulation game. These games are driven by a model which conn \t 


4 cng tom 
elements for creating such an environment exist within a simulation mode] ao 


that has been developed at ISC under Contract MDA903-78-C-2012. The ISC V/ 


a ‘ground truth' defined by an optimal winning strategy. The rudimentary 
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| simulation is called TOMM (Tactical On-Line Maneuver Modeling). The TOMM ioe id 

interactively exercises the simulation by means of a function keyboard and _ 

trackball while sitting at a four-color, vector graphics display. The user 

sets up and plays out a division-level scenario. He is responsible for sketch- 

ing terrain highlights of interest and defining opposing orders of battle. 

He specifies unit movements for both sides and, on request, the computer 

shows dynamic replays of these movements with likely sightings and engage- 

ments. The user can call for displays of detection and area of battlefield 

effectiveness contours for each unit type and size based on computer calcula- 

tions. TOMM's displays assist the user in determining reasonable travel 

routes, travel speeds: and tactical outcomes of potential engagements. Thus 

the TOMM user can analyze alternative tactical situations thereby gaining 

reasonable answers to "what if?'' questions. 

| vO 
The ''scenario generator" will be a modified version of TOMM with 


the following capabilities: 
Srna ad, ALR 
e Controllable decision anjeceives and goals 


@ Controllable environment and initial conditions 
4 @® Situation description generator 
By assigning values to a given set of parameters, the experimenter would deter-_ 


the .constraints of the simulated 


mine the nature of the mi 
environment. The assigned objective parameters would be translated into both 
a mathematical scoring function and a linguistic description (briefing) of the 
goals and their implications. Subjects will be given the time to study this 
description prior to commencing the game and, in addition, will be provided | 
with performance feedback information at various stages during the game. The 
environmental constraint parameters would determine terrain-conditions, capa- 
bility limitations, probabilistic cause-effect relationships, etc.. Some of 
these constraints will be displayed to the user at the ee of the experiment 
(e.g., terrain conditions). Others would have to be learned by the user by 
repeated experiences (e.g., probability of being detected). A built-in situa- 
tion description generator, which is model-driven, will present the user with 
graphic and textual information about the current situation, and will be up- 


dated as more information becomes available. The situation description generator 


will be driven by a probabilistic "intelligence report simulator.'' The latter 
will receive as its input the actual environmental variables (weapon capa- 
bilities, troop movements, etc.) and translate those into a set of linguistic 
statements via a well controlled, noisy channel. This can-be easily imple- 
mented by first passing the input variables through a conditional probability. 
metrix to determine the type and likelihoods of observational data obtained. 
The transformed data is then described in linguistic terms. These descriptions 


can be controlled along a number of experimental dimensions: 


Amount of information presented 
Information accuracy (noise) 
Presentation format (probabilities, curves, maps, etc.) 


Timeliness of information (age) 


Level of detail of information 


The scenario generator will have each of these dimensions formalized so that 
an experimenter can simply specify the values of key parameters to achieve 


certain global effects. 


23 EVALUATION MEASURES AND DATA ANALYSIS 


The selected evaluation measures must conform to the following require- 


ments: 


1. They must be quantitative with outcomes falling within a well- 
defined numerical scale. It is only through numerical comparisons that an 
objective measure of the effectiveness of decision support systems can be 
gained. "Expert Opinion" and ''User Satisfaction," while. valuable, can only 
give general indications of effectiveness and not the conclusive and reliable 


results that scientists usually require. 


2. They must adhere to the general objectives of the evaluation plan. 
Evaluation measures, derived from established military evaluation objectives, 
will insure that meaningful guidelines and methodology can be generated for 


future evaluation applications. 


3. They must be general enough to apply to a wide variation in cir- 


cumstances, scenarios, and decision support systems. 
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The evaluation measures briefly described below are typical of those from 


which the selection will be made. 


24.3%) Decision Quality 


Decision quality refers. to the expected improvement in performance 
through use of a DSS. This improvement can be measured by comparing the | 
performance score in similar experiments with and without the use of a DSS. 
This measurement, taken at key places along the learning curve will provide . 


valuable information pertaining to the potential application points of DSS'‘'s. 


2.3.2: Judgment Consistency 


A disagreement between directly elicited judgments (both normative and 
factual) and those derivable from other judgments clearly indicates (a) a 
deficiency in mental manipulation and (b) areas of potential benefit for DSS 
applications. The formalization of a standard measure of consistency, across 
a large number of situations and judgments would be a valuable tool for asses- 
Sing the quality of DSS methodologies. However, the elimination of inconsis- 
tencies (e.g., by ignoring a particular set of judgments in favor of another) 
would not, by itself, guarantee an increase in decision quality. The latter 


would be more directly measured by the concept of ''validity." 


2.3.3 Judgment Validity 


Judgment validity measures the degree to which a pronounced judgment 
faithfully represents the subject's experience. For example, the validity 
of an estimate of a particular event probability is measured by the proximity 
of that estimate to the actual likelihood of that event as dictated by the 
physical or mathematical model which drives the (stmulated) environment. 
Similarly, the validity of value judgments would be determined by how close 
these judgments are to the actual (maximal) expected score achievable within 


a given environment. 


2.3.4 Decision Time 


Structuring aids generally have a tendency to require a significant 


amount of time for use. It is possible to measure whether or not this time 
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expenditure is justified by the improvement in decision qulaity. We propose 
to derive a relation between the degree of structural details and the quality 
of resultant decisions. This relationship would enable us to formulate which 


level of detail would be most warranted for a given type of problem and exper- 


tise. 


2.4 EVALUATION EXPERIMENTS 


The essential principle underlying the evaluation method is that 
operational tests ought to be performed in an environment which is tightly | 
controlled by and thoroughly known to the evaluator. The merit of any decision 
plan produced by the aided user could then be measured and compared with an 


objective measure (colloquially called the ‘ground truth') of quality. To 


meet these requirements we envision the following test program: two computer 
programs would participate in the evaluation test, an environment simulator 


(scenario generator) and a decision support systém. 


The simulated environment program contains a well defined mathematical 
model of some stochastic, military problem domain. Any given choice of the | 
parameters governing the mathematical model would result in a linguistic 
description of a scenario characterizing the behavior of the environment in 
the past, as well as its reaction to the user's decisions. The user would 
be given a reasonable length of time to interact with this simulated environ- 
ment and to form a mental picture of the fabric of relations (heuristic model) | 
which determines the environment's behavior. This mental picture need not . 
have any similarity to the mathematical model which drives the simulated 
environment. However, any decision enacted by the user would result in a 
well defined (stochastic or deterministic) reaction of the environment to the 
decision, Part of the environmental model is a formula determining the user's 
score for any combination of factors which may prevail in the environment. (For 
example, user score might be expected military positional and/or material advan- 
tage at the end of a two-day peroid.) Consequently, any decision enacted by the 
user can be assigned a numerical value conditioned upon his most recent action 
and calculated on the basis of the known probabilistic parameters of the mathe- 


matical model. This objective value (utility) is occasionally displayed. to the 
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user in the training phase, and is also the function which the user attempts to 


maximize using the mental picture formed so far. 


The eperations) merit of a given decision-support system is measured 
with phased experiemtns. Once the user obtains a given level of familiarity 
with the environment (as indicated by the score for his unaided decisions), 
he will be offered the use of the aiding tools contained in the support system. 


His score with and without the aids will then be compared. 


Note that this scheme, the measured quality of the aiding technique 
is made up of three distinct activities: (1) the ability of the user to form 
a faithful picture of the simulated environment, (2) the user's ability to 
articulate his perceived picture in the format provided by the aiding machine, 
and (3) the ability of the machine to manipulate the information provided by 
the user and come up with a reasonable recommendation. Note ais that the 
user's ability to discover the underlying utility Function imposed by the 
model is as importatn as his ability to decipher the factual cause-effect | 
relationship within the environment (see Appendix B). AI}! these elements 
are importatn components in any real-life decisionmaking situation, and ought, 
therefore, to be part of the evaluation program. Moreover, having a well- 
defined mathematical model would enable us to evaluate some of these cone 
ponents separately. Fro instance, a decision-aiding tool might be found 


useful in eliciting factual-knowledge but deficient in eliciting values. 
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3.0 PROGRAM IMPLEMENTATION 


3.1 OVERVIEW 


The general objectives of the proposed program were stated in Section 
l. This section deals with the proposed program schedules, the program tasks, 


and the projected project deliverables. 


3.2 PROGRAM PLAN | 


The proposed program plan is for a two-year phased effort involving 
theoretical research and development, test bed implementation, evaluation 
experiments, data evaluation, and evaluation methodology development. The 


following subsections describe these phases in detail. 


3.261 First Year 
The phases of. the first-year effort are the Following: 


1. Develop theoretical foundations, establish verifiable hypotheses, 


and. establish quantitative performance measures. 


2. Select a suitable military application area and design the scenario 


generator for the simulated experimental environment, 


3. Implement the scenario generator test bed at the ISC facilities 


using graphic display capabilities. 


kh. Establish criteria for decision support system application and 


a taxonomy of evaluatable military operational tasks. 


5 Pe ae Second Year 
The phases of the second-year effort are the following: 


1. Select or develop a suitable decision support system for evaluation 


and perform any necessary scenario generator integration tasks. 


2. Design and conduct controlled evaluation experiments according 


to a prescribed experimental plan. 


3. Analyze and evaluate the experimental data and provide conclusions 


about decision support system merit based on the experimental results. 
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4. Develop decision support system evaluation methodology for appli- 


cation to related military aiding systems. 


3.3 MILESTONE CHART 


Table 1 shows a milestone chart for the first year program phases 
and denotes the major deliverable dates beginning with the first month after 


date of contract (ADC). 


3.4 DELIVERABLES 


The following four deliverables are projected for the proposed first- 


year effort. 


]. Theoretical Foundations Report - Describes all verifiable hypotheses, 


performance measures, and preliminary experimental plan. 


2. Scenario Generator Design Report - Describes detailed design of 


scenario generator program and includes implementation plan and program speci- 


fications. 


3. Progress Report on Scenario Generator Program Implementation - 


Describes three-months progress on program implementation. 


4. Final Report - A summary of all work completed during the first 


year. 


3.5 STAFFING 


ISC has assembled a highly qualified team to perform the proposed 
work. The team will be directed by Dr. Antonio Leal. Dr. Leal will be respon- 
sible for liaison and for ensuring that the terms of the proposed contract 
are met. All technical products will be reviewed by him to ensure that they. 


are technically sound and are responsive to customer requirements. 


ISC will also use the consulting service of Dr. Judea Pearl, Professor 
of Engineering at UCLA. Dr. Pearl's expertise in decision analysis, pattern 
recognition, and computer methods of representing complex problems to decision- 


makers will be extremely valuable to the project. 


a 
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Scenario Generator 
Design 


Scenario Generator 
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Deliverables 
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Report 
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Progress Report on Scenario 
Generator Program Implemen- 
tation 


Final Report 


Table 1. 


Months After Date of Contract 


Milestone Chart. 
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PROFESSIONAL EXPERIENCE | 


New York University Medical School Research Engineer _ 1960-61 
New York, New York 7 | a 


RCA Research Laboratories Member of Technical Staff 1961-65 
Princeton, New Jersey | | 


Electronic Memories, Inc. Director, Advanced Memory Devices 1966-69 
_.Hawthorne, California 


Consulting Activity 


| Data Industry, Hawthorne, California | 4 f69-12/69 
Electronic Memories, Inc., Hawthorne, California 3 | : 8/70-3/72 
Rand Corporation, Santa Monica, California - a 4/73-Present | 

_ Jet Propulsion Laboratory, Pasadena, California 6/74-2/76 
Integrated Sciences Corporation, Santa Monica, California : 6/75-Present 
Perceptronics, Inc., Woodland Hills, California OT 6/76-Present 


TEACHING EXPERIENCE 
Newark College of Engineering Instructor | 1961 


University of California, Los Angeles 


Computer Science Department Assistant Professor — 7/69-6/70 
- Engineering Systems Department | Associate Professor . 1/70-6/76 © 


Professor | _ 7/76-Present 
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TEACHING EXPERIENCE (continued) 


Courses Taught 


Engr. M1 Patterns of Problem Solving 

Engr. M12 Applied Patterns of Problem Solving 

Engr. 00D Information Processing Systems 

Engr.* 171A Introduction to Feedback and Control Systems: Dynamic al Control I 
Engr. 193A* Engineering Probabilistics and Stochastics 

Engr. 193B* Engineering Statistics 

Engr. 225D Computer Memories and Memory Systems 

Engr. 225M Pattern Recognition 

Engr. 271A Dynamic Systems Optimal Control | 

Engr. 2/4A* Problem Solving and Decision Making--P.S. I 

Engr. 2/74B* Problem Solving and Decision Making--P.S. II] - 
Engr. 274C* Computer Methods of Data-Analysis and Model-Formation 


* Instructor-in-Charge 


HONORS AND SPECIAL RECOGNITION RECEIVED 


RCA Laboratories Achievement Award "for development of the Superconducting 1963 | 
Parametric Amplifier” 


Performed the first experiment to prove the existence of the Magnus Force 


in superconductors, Electronic News, February 1966 1966 
NATO Senior Fellowship in Science | 1974 


Pattern Recognition Society Award for an outstanding contribution to the 1978 
Journal of Pattern Recognition (1976) | 
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Source | 


om 


Amount 


ONR NO0014-69-A-0200-4038 $ 30,000 


NSF GJ 37329 


NSF MCS 74-12208 AOl 
(formerly GJ 42732) 


NAVY. (N66001-75-C-0226) 
NSF MCS 75-18734 
ONR NOOO014-78-C-0372 


NSF MCS 78-07468 


NSF MCS 78-18924 


$ 38,900 
$ 78,000 


¢ 9,300 
$ 46,700 
$ 72,014 


$156,705 


$ 68,751 


Purpose 


Effectiveness of Orthogonal 
Transforms 


An Investigation of the Role ~ 
of Representations in Problem 
Solving | 


An Investigation of the Role 
of Representations in Problem 
Solving 


Adaptive Transforms in Signal 
Processing 


Investigating Computational 
Gains Using Partial Information 


A Goal-Directed Approach to 


Structuring Decision Problems _ 


An Investigation of the Role 
of Representations in Problem 
Solving 


Investigating Computational 


Gains Using Partial Information 
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Dates 


10/1/71- 
9/30/73 


3/1/73- 
8/31/74 


7/1/74- 


— 12/31/77 


6/1/75- 
12/1/75 


1/1//6- 
12/31/78 


5/1/78- 
10/31/79 


6/15/78- 
11/30/81 


2/15/79-. 
1/31/82 
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PUBLICATIONS 
BOOKS 


Pearl, J.s "Packing Data Tightly in Thin Film Nanoriesd in W. B. Riley (ed.), Elec- 
tronic Computer Memory Technology, McGraw-Hill Book Company, New York, 1971. | 


Pearly J., "Distinctive Properties of Quantized Vovtice: in Superconducting Films,' 
in Daunt, Edwards, Milford, and Yaqub (eds.), Low Temperature Physics-LT9, pent 
Press, New York, pp. 566- 570, 1965. 


Pearl, a > Andrews, H. C., and Pratt, W. K., "Performance Measures for Transform Data 


Coding," in H. C. Andrews (ed.), Tutorial "and Selected Papers in Digital mage 
Processing, IEEE Computer Society, pp. 173-177, 1978. 


PAPERS 


Liebman, F. M., Pearl, J., and Bagno, S., "The Electrical Conductance Properties of 


Blood in Motion," Physics in Medicine and Biology, Vol. 7, No. 2, op. 177-194, 
October 1962. 


Pearl, J., ins on Svinte RC Distributed Networks'," Proceedings of the IRE, 
Vol. 50, No. 10, October 1962. 


Pearl, J., "Current Distribution in Superconducting Films Carrying Quantized Fluxoids," 
Applied Physics Letters, Vol. 5, No. 4, pp. 65-66, August 1964. 


Pearl, J., "The Effect of Crossed Fields on the Velocity Distribution of Hollow Elec- 
tron Beams," IEEE Transactions on Electronic Devices, pp. 374-381, August 1964. 


Pearl, J. » "Optimum Synthesis of Inductorless Bandpass Filters Using a Combination of 
Identical RC Coupled Amplifiers," IEEE Transactions on Circuit Theory, Vol. CT-11, 
No. 4, pp. 457-461, December 1964. 


Pearl, J., "On the Nature of Resistance in the Mixed State of Type II Superconductors, . 
Physics Letters, Vol. 17, No. 1, pp. 12-14, June 1965. _ 


Pearl, J., "Direct Evidence of Steady EMF Induced by Flux Motion in Superconductors ," 
Physical Review Letters, Vol. 16, No. 3, pp. 99-100, January 1966. 


Pearl, J., "Moving Vortices in Superconductors," Electronics, June 13, 1966. 


Pearl, J., "Electromagnetic Behavior of Thin-Film emer uneese Journal of Applied 
“Physics, Vol. 37, No..8, pp. .2956-2963, July 1966. 


Pearl, J., "Structure of Superconductive Vortices Near a Metal-Air Interface," Journal 
of Applied Physics, Vol. 37, No. 11, pp. 4139-4141, October 1966. 


Pearl, J., "Field Distribution of Plated Wire Memory Elements and Its Effect on Memory 


Characteristics,’ IEEE Transactions on Magnetics, Vol. MAG-4, No. 3, p p. 366-371, 
- September 1968. 3 | a — 
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PAPERS (continued) 


Pearl, J., "Application of Walsh Transform to Statistical Analysis," IEEE Transactions 
on Svstems, Man and Cybernetics, Vol. SMC-1, No. 2, April 1971. 


Pearl, J., "Walsh Processing of Random Signals," eee Transactions On: Electromagnetic 
Compatibility, Vol. EMC-13, No. 3, August 1971. 


Pearl, J., "Basis-Restricted Transformations and Performance Measures for Spectral 
Representations,” IEEE Transactions on Information Theory, Vol. IT-17, No. 6, pp. 


orca aH tte Daath arose ape Nsnkcastah saints rERSOrSAPOSSnseSETRIAt SA eTINe tt RSAC ED 


731- ia November 1971. 


Pearl, oe. "The Hilbert Schmidt Decorrelation Measure for Fourier and Walsh Transforms," 


Proceedings of the Symposium on Applications of Walsh Functions, March 1972 
(AD-744650) . | 


Pearl, J., Andrews, Harry C., and Pratt, William K., "Performance Measures for Transform 


Data Coding," IEEE Transactions on Communication Technology, Vol. COM-20, No. 3, pp. 
411-415, June 1972. | 


Pearl, J., "Time, Frequency, Sequency and Their Uncertainty batacions,* IEEE Transac- 
tions on Information Theory, Vol. IT-19, No. 2, pp. 225-229, March 1973. 


Pearl, J., "On Coding and Filtering Stationary Signals by Discrete Fourier Transforms," 
TEEE Transactions on Information Theory, Vol. IT-19, No. 2, pp. 229-232, March 1973. 


Pearl, J. and Saltzman, J., ‘Comparison of Fourier and Walsh Transforms in Radar Signal 


Processing, " Proceedin S ‘of the Symposium on Applications of Walsh Functions, April 
1973 (AD 763000). | | | 


“Mgazi, V. R., Fino, B. J., and Pearl, J., “Comments on ‘Basis-Restricted Transforma- 
tions and Performance Measures for Spectral Representations’ (Corres.)," IEEE Trans- 


actions on Information Theory, Vol. IT-14, No. 4, pp. 564-565, July 1973. 


Pearl, J., "Storage vs. Error Tradeoffs in Pattern Recognition and Classification," 
Proceedings of the Conference on Computer Graphics, Pattern Recognition, and. Data 
Structure, IEEE Cat. No. 73CHO981 ~ 1C, Los Angeles, Pay Onwes pp. 239-342, May 14- 
16, 1975. 


Pearl, J., "Optimal Dyadic Models of Time-Invariant Systems," IEEE Transactions on 
Computers, Vol. C-24, No. 6, pp. 598-603, June 1975. 


Hamidi, Massih and Pearl, J., "On the Residual Correlation of Finite-Dimensional 
_ Discrete-Fourier-Transforms of Stationary Signals," IEEE Transactions on Information - 
Theory, Vol. IT-21, No. 24, pp. 480-482, July 1975. 


Pearl, J., "On the Storage Economy of Inferential Question-Answering Systems," IEEE 
Transactions on Systems, Man and Cybernetics, Vol. SMC-5, No. 6, pp. 595-602, 
November 1975. | | 
Pearl, J., "On the Complexity of Inexact Computations," Information Processing Letters, 
Vol. 4, No. 3, pp. 77-81, December 1975. ; 
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PAPERS (continued) 


Pearl, J., "Asymptotic Equivalence of Spectral Representations," IFEE Transactions on 


Acoustics, Speech, and Signal Processing, Vol. ASSP-23, No. 6, pp. 947-5951, 
~ December 1975, 


Pearl, J., "Qn Coding Precedence Relations with a Pair-Ordering Fidelity Criterion," 
TEFE Transactions On Information Theory, Vol. IT-22, No. 1, pp. 118-120, January 
1976. 


~ Pearl, J., "An Application of Rate-Distortion Theory to Pattern Recognition and 
Classification," Pattern Recognition, Vol. 8, No. 1, pp. 11-22, January 1976. 


Pearl, J., "Memory vs. Error-Characteristics for Inexact Representations of Linear 
Orders," IEEE Transactions on Computers, Vol. C-25, No. 9, pp. 922-928, September 
1976. — 


Pearl, d.; "Theoretical Bounds on the Complexity of Inexact Computations," JEEE Trans- 
actions on Information Theory, Vol. IT-22, No. 5, pp. 580-586, September 1976. 


Pearl, J., “On the Complexity of Imprecise Causal Models," IEEE Transactions on Systems, 
Man and Cybernetics, Vol. SMC-6, No. 9, pp. 652-655, September 1976. 


Hamidi, M. and Pearl, J., "Comparison of the Cosine and Fourier Transforms of Markov-1 


Signals," IEEE Transactions on Acoustics, Speech and Sena Haske aE Vol. ASSP-24, 
No. 5, pp. 428-429, October 1976. 


Pearl, J., "A Framework for Processing Value Judgments," IEEE Transactions on Systems, | 
Man and Cybernetics, Special Issue on Behavioral Decision Making, Vol. SMC-7, No. 5, 
‘PP. 349-354, May 1977. 


Leal, A. and Pearl, J.; “An Interactive Program for panvemacional Elicitation of 
Decision Structures," IEEE Transactions on Systems, Man and Cybernetics, Special 
Issue on Behavioral Decision Making, Vol. SMC-/, No. 5, pp. 368-376, May 1977. 


Pearl, J., "A Note on the Management of Probability Assessors," IEEE Transactions on 
Systems , Man and Cybernetics, Special Issue on Behavioral Decision Making, Vol. 
SMC-7, No. 5, pp. 402-403, May 1977. 


Pearl, J., "On Summarizing Data Using Probabilistic Ascegcione” IEEE Transactions on 
Information Theory, Vol. IT-23, No. 4, pp. 459-465, July 1977. 


Pearl, J., "Capacity and Error-Estimates for Boolean Classifiers with Limited seiicee 


| ity," Proceedings of the IEEE Computer Society Conference on Pattern Recognition 
and Image Processing, May 30-June 2, 1978. 


Pearl, a "An Economic Basis for Certain Methods of Evaluating Probabilistic Forecasts," 
International Journal of Man-Machine Studies, Vol. 10, pp. 175-183, 1978. 


Pearl, J., "On the Connection Between the Complexity and Credibility of Inferred pOGeeeE 
International Journal of General Systems, Vol. 4, pp. 255-264, 1978. 
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PAPERS (continued) 


Crolotte, A. and Pearl, Jd., “Asymptotic Rate-Distortion Functions for Coding Precedence 
Relations," IEEE Transactions. on Information Theory, Vol. IT-25, No. 1, pp. 80-82, 
January 1979. 


Crolotte, A. and Pearl, J., "Bounds on Memory Versus Error Trade-Offs in Question- 


Answering Systems," IEEE Transactions on Information Theory, Vol. IT-25, No. 2, 
pp. 193-202, March 1979. 


ACCEPTED FOR PUBLICATION 


Pearl, J., "Entropy, Information and Rational Decisions,’ invited paper, to be published 
in the International Journal of Policy. Analysis and Information Systems, Special Issue 
on "Mathematical Foundations and Models of Socio-Economic Systems", Vol. 3, No. 1, 
July 1979. 


Crolotte, A. and Pearl, J., "Elasticity Conditions for Storage vs. Error Exchange in 
Question-Answering Systems," to be pense in IEEE Transactions on Information Theory. 
November 1979. 


Pearl, J., "Capacity and Error-Estimates for Boolean Classifiers with Limited tonientinst 
to be published in IEEE Transactions on Pattern Analysis and Machine Intelligence. 


Pearl, J., "Toward a Computational Model of Visual Thinking," to be published in 
International Journal of Man-Machine Studies. 


Yemini, Y. and Pearl, J., “Asymptotic Properties of Discrete Unitary Transforms," to be. 
published in IEEE Transactions on Pattern Analysis and Machine Intelligence. 
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TECHNICAL REPORTS (From 1970, unpublished) 
Pearl, J., "The Mystery of Problem Representations," UCLA-ENG-0572, March 1972. 


~ Pearl, J., "On Demonstrating Effectiveness of Decision Analysis," Jens ENG-i2/2, 
September 1972. 


Pearl: J., "Effectiveness of Orthogonal Transforms in Information Processing ewe 
tions," UCLA-ENG-7387, November 1973. | 


Pearl, J., "Problem Representation Research," UCLA-ENG-7404, December 1973. 


Pearl, Je. "Demonstrating Benefit of Training Toward Realistic Assessment of Uncertainty," 
Rand Internal Note IN-22803-ARPA, January 1974, UCLA-ENG-PAPER-O1/5. 


Pearl, J., "Maximum Likelihood Estimation of Realism Curves," Rand Internal Note IN- 
22804-ARPA, January 1974, UCLA-ENG-PAPER-0275. 


Pearl, J., "On the Management of Probability Assessors," and "Some Information-Theoretic 
Results Concerning the Ranking of Inquiries by Forecasters Employed Under Logarithmic 
Scoring Rules," UCLA-ENG-PAPER- 0375, February 1974. 


Pearl, J., "Error-Bounds for Inferential Question-Answering Systems with Limited Memory," 
UCLA-ENG-7481, October 1974. 


Pearl, d., "Investigating Adaptive Transforms in Information Processing Applications," 
~ UCLA-ENG-7567, December 1975. 


Pearl, J., "Toward a Computational Model of Visual Thinking," UCLA-ENG-7788, a 
1977. | 


Pearl, J., "Capacity and Error-Estimates for Boolean pede Sunehe with Limited Complexity," 
UCLA- ENG-7/789, December 1977. 


PATENTS 


Apparatus, Without Moving Parts, for Moving #35201 ,765 © August 17 ,1965 
a Storage Area Along a Storage Medium | 


Cryogenic Heat Pump Including a Magnetic Means #35393,526 July 23, 1968. 


for Moving a Normal Zone Along a Super- 
Conductive Rod | 
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— CONTRIBUTIONS TO CONFERENCES, SEMINARS AND SYMPOSIA (from 1974) 


Pearl, J., “Process-Model for Learning Realistic Assessments of Probabilities,” 
presented at the Conference on Bayesian Research, Los Angeles, California, February 
~—625-26, 1974. 


Pearl, J., “On Two Problems of Knowledge Representation: Quantitative Coding of Un- 
certainty and Problem-Space Transformations," a seminar presented at the School of 
Artificial Intelligence, University of Edinburgh, Edinburgh, Scotland, July 15, 1974. 


Pearl, J., "Research in Knowledge Representation," a seminar presented for the UCLA. 
7 Computer Science Department, UCLA, Los Angeles, California, November 7, 1974. 


Pearl, d. . Oras and Technical Coordinator of Workshop on Computer Rapresew ation 
of Knowledge, UCLA, Los Angeles, California, December 4, 1974. 


Pearl, J., "Storage vs. Error Tradeoffs in Pattern Recognition and Classification," 
presented at the Conference on Computer Graphics, Pattern Recognition and Data 
Structures, Los Angeles California, May 14-16, 1975. | 


Pearl, J., "Complexity of Inexact Computations," a seminar presented for the UCLA 
Computer Science Department, Los Angeles, California, June 5, 1975. Also presented 
at the University of California, Berkeley, July 15, 1975. | 7 | 


Pearl, J., “Economic Basis of Probability Scoring Rules," a Seminar presented at the | 
Stanford Research Institute, Stanford, California, July 16, 1975. Also presented at 
the Sucial Science Research Institute, University of Southern Cay iTOrnlas Los Angeles, 
California, July 25, 1975, 


Pearl, J., "On the Storage Economy of Error-Tolerating plies ion=fnewaring Sehane 8 
presented at the International Joint Conference on Artificial Intelligence, Tbilisi, 
USSR, September 3-8,. 1975. 


Pearl, Jd. and Leal, A. "A Domain-Independent System for Interactive Elicitation of 
Problem Structures," presented at the International Joint Conference on Artificial 
Intelligence, Tbilisi, USSR, September 3- 8, 1975. 


Pearl, J., "On the Complexity of Inexact Information a ee at the 9th 
Hawaii International Conference on System Science, Honolulu, Hawaii, January 8, 1976. 


Pearl, J., “Artificial Intelligence and Decision Analysis," presented at the 1976 
| Conference on Bayesian Research, Los Angeles, California, February 1976. 


Pearl, J., "On the Complexity of Descriptions," presented at the IEEE Workshop on Data 7 
Structure and Pattern Recognition, Albuquerque, New Mexico, February 11-13, 1976. 


Pearl, J., “Theoretical Bounds on the Complexity of Inexact Computations," a seminar 
presented at the Technion, Israel Institute of Technology, Haifa, Israel, November 
23, 1976. 


Pearl, J., "On the Connection Between the Complexity and Plausibility of Inferred 


Models," a seminar presented at the Department of Applied Mathematics, Weizmann 
Institute of Science, Rehovot, Israel, January 17, 1977. : 
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CONTRIBUTIONS TO CONFERENCES, SEMINARS AND SYMPOSIA bocse 1974) (continued) 


Pearl, J., "Theoretical Bounds on the Complexity of Inexact cinenatattons,* a seminar 
presented at the Hebrew University, Jerusalem, Israel, January 24, 1977. 


Pearl, J., “Asymptotic Properties of Orthogonal Transforms with Applications to 
Digital Signal Processing," a seminar presented at the Department of Mathematics, 
Tel ‘Aviv University, Tel Aviv, Israel, January 31, 1977. 


Pearl, J., "Approximate Diagonalization of Toeplitz Matrices with Applications to 
Digital Signal Processing," presented at the Department of Mathematics, The Weizmann 
Institute of Science, Rehovot, Israel, February 8, 1977. 


Pearl, J., “On the Connection Between the Complexity and Plausibility of Inferred | 
Models, ; a seminar presented at the International Conference on General systems 
Research, PUNGHeMe Ons New York, August 1977. 


tear: A. and Pearl, Jes "An Interactive System for Conversational Eieieation of 
Decision Structure," presented at the 5th International Joint Conference on Artificial | 
Intelligence, Boston, Massachusetts, August 12-25, 1977. 


Pearl, J., "On the Connection Between the Complexity and Plausibility of Inferred 
Models," presented at the Jacob Marschak Interdisciplinary Colloquium on Mathematics 
in the Behavioral Sciences, Los Angeles, January 13, 1978. 


Pearl, .., "Are Simpler Ruler More Trustworthy?", presented at the 14th Conference on 
Bayesian Research, Los Angeles, California, February 1978. 


Pearl, J., "Domain Independent Decision-Aiding Systems ." seminar presented at the 
Lawrence Berkeley Laboratories, March 22, 1978. 


Pearl, Je; "Toward a Computational Model of Visual Thinking," seminar presented at. | 
the Department of Psychology, Stanford University, and at Stanford Research Institute, | 
Stanford, California, March 23, 1978. 


Pearl, J., "Capacity and Error Estimates for Boolean Classifiers with Limited Complexity," 
presented at the IEEE Computer Society Workshop on Pattern Recognition and Artificial 
Intelligence [also Panelist on Common Methodologies], Princeton a Princeton, 
New Jersey, April 12-14, 1978. 


Pearl, a and Leal, A., "An Interactive Model-Structuring Program for Coaching a Panel 


of Experts," Symposium on Modelling and Simulation Methodology, The Weizmann Institute 
of Science, Rehovot, Israel, August 13-18, 1978, 
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4,0 ASC QUALIFICATIONS 


Le] CORPORATE BACKGROUND 


Integrated Sciences Corporation was established in 1967 by a group 
of experienced researchers to bring their diverse skills and backgrounds to 
bear on a wide range of governmental and industrial system problems. Company 
operations began with a staff of five professionals, on two government con- 
tracts. The company has grown steadily since that time, and over 170 contracts 
have been completed. The expertise of ISC personnel has been successfully 
applied to problems of military and civil systems, including energy, trans- 


portation and environmental planning. Research areas include work on: 


Mathematical Modeling and Simulation 

Computer Graphics and Display Engineering 

Man-Machine Interface Design 

Conceptual and Functional Design of 

Technological Systems 

e Decision Support Systems 

@® Test/Experiment Design and Statistical 
Analysis of Data | 

@ Econometric and Cost Effectiveness Analysis 

Program Planning and Monitoring 

@ Applications of Systems Analysis and 
Operations Research Techniques 

e@ Analysis of Management Eventing) Decision 

yeti 


a 


Much of ISC's past work has resulted from referrals by satisfied 
customers. Clients whom we have served include the Naval Ocean Systems Center, 
Hawaii and San Diego Laboratories; the Office of Naval Research; the Army 
Research ineeitaves the Naval Underwater Systems Center; the Naval Material 
Command; the Naval Sea Systems Command; the Naval Ship Engineering Center; 
the Naval Underwater Systems Center; and the Naval Coastal Systems Center, 


Panama City. 


ISC's full-time professional staff currently consists of engineers, 
scientists, mathematicians, computer programmers, and experimental psycholo- 
gists. Work is currently in process on fifteen contracts for different 
government agencies. A sample of current and recent work performed by ISC 


is given below: 
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Development and experimental evaluation of algorithms 
and man-machine interfaces that aid Naval task force 
commanders in selecting optimum tactics for penetrating 
enemy defenses. 


Development of interactive graphics software and man- 
machine interfaces used by an Army field commander 

to simulate hypothesized enemy tactics. This aid 
enables the user to test his hypotheses of enemy tactics 
against observed intelligence data. 


Applied research to investigate the benefits of color- 
coded displays and related fire control algorithms for 

a new submarine combat system. Includes simulation of 

the combat system on ISC's Simulation and Display Facility 
equipment and the running of a human factors experiment. 


Performance evaluation, through computer modeling and 
simulation, of several new, terminally guided, high rate, 
neutralizer concepts for countering ocean. mines. 


Development and use of a computerized model that 
simulates swimmer attacks against assets defended by 
multiple sensors and weapons. 


Design of a display interface between an operator 
and a computer that processes data From elements of 
a complex defense system. : 7 


Development of a proprietary new method for using 
standard sonar outputs to construct target images for 
deep ocean sonar systems. Includes simulation of the 
sonar's performance and the operation of the image 
construction and tracking algorithm. | | 


Development of an analytic method for using group sub- 
jective judgment data to prioritize exploratory 

development budgets. Included is the design of interactive 
graphics (computer) software enabling managers to use the 
stored subjective judgment data to explore the. conse- 
quences of alternative funding allocation phi losophtes. 


Development and use of a computerized search/theoretical 
model to optimize the search phbateoy of a system seeking 
underwater targets. 


Economic and operations analysis of Sr eeenabive concepts 
for growing kelp in the open ocean, harvesting and pro- 
cessing the kelp into natural gas and other products. 
Includes economic evaluations of subsystems, analysis 

of transportation trade-offs and system design. 


Preparation of a comprehensive plan for a twenty~-million- 
dollar marine biological sciences program. 
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@e Formulation and use of a method for obtaining and 
using subjective judgment data on multiple and diverse 
evaluation factors to select the best of alternative 
systems. | 


42) MANAGEMENT APPROACH 


ISC management believes that definition of complex problems and 
development of approaches for their solution require the application of 
several scientific disciplines—an integration of sciences with their 
respective fields of knowledge and technology. Therein lies the premise 
for our name and the basis for our management philosophy. We utilize the 
systems approach to problems. Careful consideration is given to all aspects 
of the subject system, interfacing systems and the environments within 
which the system must operate to insure that meaningful and useful solutions 
are obtained. Also, we are quite sensitive to the need for practical 
approaches and designs that consider both material and time constraints and 


limitations of resources. 


Accordingly, at the beginning of a proposal effort, ISC's President 
designates as project/contract manager a professional with an established 
reputation In the subject area, and the ability to communicate effectively 
with the customer and lead project personnel. The contract manager sub- 
‘divides the work into task areas and, subject to the President's concurrence, 
selects senior staff members with the skills and experience needed to lead 
work in each task area. The contract manager and task leaders organize a 
team with all necessary disciplines appropriately represented, and assign- 
ments are made on the basis of expertise, past experience, capability and 


project requirements. 


In the first week of the contract, responsibilities and time schedules 
are detailed to ensure proper interface with other research projects. Task 
leaders maintain close contact with researchers, holding project meetings with 
all team members and briefings for the project manager about once every week, 


or more frequently if required. 
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Periodic meetings are scheduled with staff personnel and consultants 
to ensure proper orientation of effort at each stage of the research. This 
enables us to evaluate research plans and accomplishments of each task and 
make appropriate changes where necessary to avoid costly duplication and main- 


tain a close orientation toward project goals. 


In our research projects we maintain close contact with the sponsor's 
contract monitor to keep him apprised.of all research developments and resolve 
any problems that may arise. The ISC contract manager assumes the primary 


responsibility for proper liaison with the contract monitor. 


4.3 SIMULATION AND DISPLAY FACILITY (SDF) 


ISC's computer and display facility at its new offices was designed 
for command and control research from the computer floor up to the separately 
dimmed lighting fixtures. The facility has 550 square feet of space and ISC 
uses a Varian 73 minicomputer and an IDIIOM four-color, refresh, vector 
graphics display as vehicles for performing research. A block diagram of 
the system is shown in Figure 2.. The system is highly expandable at low 
cost per unit of increased capability. For example, the present computer/ 
display interface equipment can support four displays identical in capability 
to the one ISC has now; the Varian can be expanded to 256K of core, time 
sharing software, and multi-processing. Further, the Varian machine has 
already been interfaced with a sophisticated color raster display system | 
(RAMTEK), thus this capability is easy to add if desired. The Varian and 
the IDIIOM display are described below. © | _ 


1. Varian 73 


The Varian 73 minicomputer processes l6-bit words with a full cycle 
time of 660 nanoseconds for the core memory. The CPU contains 16 general . 
purpose registers. Input/Output between the computer and the peripherals 
(other than the IDII0M/!1) is handled via the Direct Memory Access (DMA). 
This minimizes the burden placed on the CPU by the peripherals, IDIIOM/I1 
input/output is accomplished via a dedicated Priority Memory Access (PMA) 


channel. The PMA transfers data between the memory and the display at over 


52" 
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Card Reader — 32K Core 
Memory 


High Speed 
Paper Tape 
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Keyboard 
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Figure 2. Hardware Configuration Diagram, © 


one million words/second. This precludes the data rate from interfering with 
display performance. The instruction set consists of 159 instructions, single 
and double word. The CPU features hardware multiply/divide and a real time 


clock. Eight levels of hardware priority interrupt are implemented (64 are 


possible). 
Be Display Hardware 


The IDISOM/11 display is a four-color, large-screen graphics system. 
The IDIIOM/11 consists of a CRT along with its associated logic and contro] 
hardware. This system is driven by the Varian 73. The IDIIOM is highly 
interactive iivroush the use of the light pen, trackball, joystick, function 


keyboard, and alphanumeric keyboard. 


The IDIIOM/I1 uses a common core memory which can be independently 
and simultaneously accessed by two processors. One of these is the Central 
Processing Unit (CPU) of the minicomputer which acts as the Display Processor. 
The other processor is a special purpose Display Generator which is an integral 
part of the IDIIOM itself. The CRT is refreshed from this memory to provide 
a non-flickering display. About 6,600 characters, 5,500 short line segments, 


or 15,000 inches of long line segments can be displayed. 


An additional capability is that of virtual display. This permits 
a display "universe'' consisting of 4,096 "Worlds'' to be drawn. The display 
screen itself is one "world."' The use of relative positioning allows the 


display window to be moved anywhere within these many "worlds." 


Four intensity levels are also available under program contro}. Any — 
graphic element can be blinked under program control. Four vector and circle 
line structures are available under program control: solid, dash, dot, and | 
dash-dot. Also, characters can be rotated 90° counter-clockwise and plotted 


vertically under program control. 


The CRT is a 21" diagonal penetron with a four color capability: red, 
Orange, yellow, and green. The faceplate is HEA coated and etched for high 
contrast. The phosphor is P22. Average time to change color js fifteen psec. 


Spot size is .015'' to .025'' depending upon intensity. 
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4k RELEVANT EXPERIENCE AND QN-GOING WORK 


Integrated Sciences has specialized during the last five years in 


developing decision support systems that use: 


Interactive graphics software 
Simulation models, resource allocation algorithms, and weapon 
trajectory algorithms that are exercised interactively by 


decision makers at displays 


ISC is on the cutting edge of the state-of-the-art in these areas. Despite 
ISC's modest size, several of its products in these areas are more sophisticated 
than similar products developed by the giants of the aerospace industry and 

the giants of the defense studies and analysis industry. We use our simulation 
and display facility (SDF) in the accomplishment of most of our work. Our 

SDF was established in 1974. It contains a Varian 73 minicomputer and a 

fully interactive, four-color, high speed IDIIOM I! display system. The SDF 


was set up to support research and development for command and control systems. 


This section summarizes major project experience relevant to the re- 
quirements of the RFP. Members of the proposed staff for the RADC work have 
directed or participated in each of the projects named. Subsections 3.1 
through 3.8 are written so as to emphasize our experience in developing inter- 
active graphics software, simulation models, resource allocation algorithms, 
and weapon trajectory algorithms. Function and task analysis of the kind 
required by the work statement were part of the work performed for each of 
these projects but function and task analysis were not the primary outputs 
of the work. Subsections 3.9 and 3.10 are summaries of projects for which 
function and task analysis were major products of our design work for complex 


systems. 


4.4.1 > Operational Decision Aids For Task Force Commanders 


| ISC has been working continuously for the Office of Naval Research 
under Contract N0O0014-75-C-0811 since March 1975 in the area of tactical 
decision aiding. ‘The contract title is Decision Aid Interface!’ and the value 


of the contract to date is $344,800. Under this contract ISC participates as 
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a member of the ONR Operational Decision Aids (ODA) team to investigate 
interactive computer graphics aids that a Naval task force commander can use 
in performing his functions in a real-time operational environment. The ONR 


contract monitor is Dr. Martin Tolcott (telephone: 202 - 696-4506). 


ISC has already designed and experimentally evaluated three inter- 
active graphics decision aids under this contract and is currently in the 
process of performing this same set of tasks for a fourth decision aid. In 
each case we have used UCLA students (mainly science and engineering students) 
as subjects in our simulation and display facility. One element of the 
work involved the implementation and experimental evaluation of an aid called 
"operator aided optimization’ (OAO). This type of aid enables the user to 
make inputs to a machine which operates on the inputs and shows problem solu- 
tions derived from the inputs. The user and the machine work iteratively 
until the user is satisfied with the machine solution. Problems for which 


OAO is appropriate tend to have the following characteristics: 


1. Solution space is of high dimensionality (e.g.,> 5); 
2. Criterion function is nonlinear, and multi-modal, therefore 
the machine working alone may only find a local optimum 


instead of the global optimum. 


The problem for which the ISC aids were constructed is the selection 
of (a) an air strike path through a field of ten enemy sensors and (b) aircraft 
speeds on each leg of the path. Utility of each candidate strike path was 
computed according to a predetermined utility function; the utility function 
was nonlinear and multi-modal. One of the ISC-designed aids uses a nonlinear 
programming (NP) algorithm; the other uses a dynamic programming (DP) algorithm. 
Figure 3 is a photograph of a typical CRT display format presented to the 


operator during the interactive solution process. 


An experiment was designed by ISC to compare decision performance for 
both types of aids. Sixteen UCLA students solved problems with and without © 
the aid and these data were compared. Subject data using the aids were also 


compared with the data resulting from using the algorithm in the automated mode, 


The principal findings from the experiment were: 
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Figure 3. Typical Display Format Presented to 
Operator During the [Interactive Solution 
Process for Defense Penetration Analysis 
for Carrier Air Strikes. 
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1. The operators using the NP aid did significantly better than 
without the aid. The average improvement across all subjects 
and trials was 29% with a range of 9% to 123%. Performance 
was significantly different across operators but this was solely 
for unaided operation. Thus the aid served as an “'equalizer." 
It enabled operators having relatively low scores without the 
aid to do as well as those who had relatively high scores 
without the aid. | | 

2. Operators using the DP aid did significantly better than wi th- 
out the aid. The average improvement across all subjects and 
trials was 12% with a range of 3.5% to 27%. 

3. The lack of a technical education was apparently not an impedi- 
ment to good performance with or without either aid. (Some 
of the operators did not have a technical education.) 

4. The average time required to adequately train an operator to 
use either aid was about four hours. | 

5. A potential implication of the findings is that OAO is attrac- 


tive to use when it is applicable because: 


a. The operator can see what is happening during the optimi- 
zation. With pictorial problem representation, he can 
make adjustments to the optimization process or results to 
compensate for limitations in problem representation more 
easily than when there is no pictorial representation. 

b. The time required to train operators to use OAO with pic- 
torial problem representation is apparently relatively short 
and does not require technical knowledge of the optimization 


algorithms. 


4.4.2 Operator Aided Cruise Missile Trajectory Computation Technique 


Under Contract N00140-78-C-6141 with the Naval Underwater Systems 
Center, Newport, Rhode Island, ISC performed research and analysis for develop- 
ing new operator-aided cruise missile (CM) trajectory computation techniques 
for advanced underwater combat systems. The dollar amount of the contract 
was $55,000 and the contract monitor was Mr. Tony Bessacini, Code 3522 7 
(telephone: 401 - 841-4576). The work was performed from December 1977 to 
December 1978. | 
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The overwater control problem which is interactively solved by the 


aid we developed is stated as follows: 


Select a set of waypoints which will delineate a series 

of constant velocity cruise missile trajectory legs so 
that the cumulative probability of detection through a | 
given field of sensors is minimized as the missile travels 


from its launch point to the landfall point. 


The enemy sensor field which must be penetrated includes both stationary sensors. 


and sensors whose locations are uncertain and therefore must be modeled by 


probability distributions. One important achievement of our work is that we 
were able to mathematically represent sensors with uncertain locations In a 
way that enabled the optimization algorithm to find the best penetration path 
in_a computationally efficient way. (This is an exceptionally difficult prob- 


lem which no one outside of ISC had solved up to the time that we did it.) 


Our aid enables the operator to do the following using our interactive 
graphics display: | 


1. Enter a hypothetical track of future submarine movement and 
pick a potential future cruise missile Taunch point and a 
landfall point. | 

2. Sequentially use dynamic and nonlinear programming algorithms 

to find the best penetration trajectory from the chosen launch 
point. Our technique takes advantages of the strengths and 


avoids the weaknesses of each optimization type. 


Figure 4 shows a field of seven stationary sensors, four of which are 
on a coastline and three of which are off the coastline. The rectangle repre- 
sents a sensor whose position within the rectangle is uncertain. The four- 
‘segment path on the right side of the figure represents the submarine's 
- potential future track. The hypothetical launch point ts on the second leg 
of the submarine track, assuming that the ship is moving from south to north. 
The launch point is connected to the designated landfall point by an initial 


trial solution and the final optimum trajectory. 


=39- 


SOLUTION, 


“VAAN Time 10 7 
nO. LAN 


$3 


RONG 


Figure 4. Scenario, Projected Own Ship Track, and Alternative 
Cruise Missile Trajectories to the Landfall Point. 


4.4.3 Interactive Modeling of Army Battlefield Tactical Events (1977-1978 
Contract 
ISC has been working for the Army Research Institute since April 1977 
to develop interactive graphics aids for planning battlefield tactics. The 
work described in this subsection was done under Contract DAHC19-77-C-0022; 
contract value was $60,563 and the work was performed from April 1977 to 
February 1978. The contract monitor was Dr. Stanley Halpin (telephone: 


202 - 274-9045). 


We developed a modeling methodology, implemented it in software on 
our simulation and display facility, exercised the software, and carried out 
a preliminary evaluation of the methodology. The display software presented 
the operator/trainee with a geographical display of the battlefield area 
highlighting the relevant topographic and cultural] features. In addition, the 
background software simulated the movement of actual enemy units as prescribed 
by a selected scenario containing a specific enemy order of battle. The actual 
enemy order of battle and movements were not shown to the operator. The 
operator/trainee selected the number, types, and routes of intelligence- 
gathering ground and air patrols. Then, as a function of the operator/ 
trainee selected intelligence gathering tactics, a certain sequence of contact 
and observation reports was made available to him. From the sequence of 
reports generated by these patrols (which were determined by probabilistic 
detection and observation functions) the operator/trainee could interactively 
establish a hypothetical enemy order of battle and tactics scenario. Ina 
real time context he could then evaluate his hypothetical solution so that it 
explained in the best way possible the continuing stream of intelligence reports 
input to him by the patrol units he dep loyed. This system is designed to 
evaluate the operator/trainee's ability to integrate a large amount of spatially 
and temporally varying data, and draw an accurate estimate of the enemy order 
of battle and disposition. Figure 5 presents a typical display shown to the 
operator/trainee during his analysis process. {SC has begun the second phase 


of this work for ARI; the new work is described in Subsection 4.4.4. 


aie 


Figure 5. 


i 
ri 
% 


ene: 


~& 


Typical CRT Display to Operator of 
Army Tactical Planning Aid. 


4.4.4 =Interactive Modeling of Army Battlefield facrreal Events (1978-1979 
Contract) RR es ea ERT EP Ue eS Snakes 


Contract) 


The current Army Research Institute work ts being done under contract 


MDA 903-78-C-2012; contract value is $66,984; contract term is July 1978 to 


July 1979. The contract monitor is Dr. Franklin Moses (telephone: 202 - 


274-9045). 


The system we are developing is called TOMM (Tactical On-Line Maneuver 


Modeling). Three elements of TOMM are being developed under the current con- 


tract. These are: 


le 


"tL ook-Ahead!! Battlefield Model. This model enables the user 


to interactively define orders of battle for both sides, to 
designate a movement cluster of units and to move the cluster 
according to one of several predefined rules. The model also 
contains rules for detection and area of battlefield ef fective- 
ness (AOBE) of each unit type and size. The user can call 
routines to display detection and AOBE contours. The look- 
ahead model also simulates engagements and graphically displays 
engagement results. The model and man-machine interface are 
structured so that the user can rapidly. analyze alternative 


tactical situations thereby obtaining reasonable answers to 


"what if?" questions. 


: Terrain Realism Model. The model allows the user to define 


forests, hills, cities, roads, lakes and rivers. Terrain 
mobility for each unit type is displayed in terms of GO, SLOW- 
GO, VERY SLOW-GO, and NO-GO regions as a function of the move- 
ment rate for the designated unit type. - 
Battlefield Representations. ISC is developing software that 
will display: | 
(a) Static contours showing where units of given size/type 
can reach in future time when they start from a given 
_place. Figure 6 indicates what this display will show. 


(b) Dynamic movement of a future-position contour. 
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Figure 6. Display of Contours Showing a Designated 
| Unit's Potential Future Positions. 
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hus Design and Test of An Interactive Computer Graphics system For Post- 
Launch Control of Submarine Launched Torpedoes 


Under Contract N00140-76-C-6272 with the Naval Underwater Systems 
Center, Newport, Rhode Island, ISC designed, implemented and is now operating 
a complete Weapon Order Generator (W0G) console of a new submarine fire control 
system. The dollar value of the contract was $39,214; the work was done from 
December 1975 to January 1977. The contract monitor was Mr. Tony Bessacini, 
Code 3522 (telephone: 401 - 841-4576). 


The WOG portion of the overall FCS is instrumented in the ISC Simula- 
tion and Display Facility and includes all the support software to simulate 
the FCS informational environment and on-line measurement performance data. 
The simulation calculates and continuously displays data in graphic and alpha- 
numeric formats on the following target submarine and own-torpedo variables 
as they change during a problem: position, course and speed in three dimensions. 
The values of the displayed variables differ from the ''true'’ values by errors 


that would be present in a real encounter. 


An experiment was conducted. Each subject's task (there were seven 
subjects) was to control the torpedo in 72 trials in which the following were 


varied: 


e@ Weapon control mode (2 levels, Corrected Intercept and 
Bearing Rider) | 
@ Man-Machine interface design (2 levels, monochromatic and 
four-color) 


-@ Target-to-torpedo geometries (3 levels). 


A set of critical operator decisions and system parameters was automatically © 
recorded by the computer as each trial progressed. A five-way analysis of 
variance was performed on the effects of weapon control mode, MMI design, 
target-to-torpedo geometries, subjects, and replications on system performance. 
One conclusion of the study is that use of color displays improves system | 
performance in some situations and degrades performance in others. Therefore, 


system designers should not use color indiscriminately. 


Shee 


Figure 7 shows the display format during execution of interactive con- 
trol of the simulated torpedo. The line emanating from the right center of 
the screen is the current estimated bearing to the target. The rectangle bi- 
sected by the bearing line represents tlo estimated error in the Kalman filter 
solution for the submarine's location. The current estimated target location 
is at the center of the rectangle and estimated target course and speed is 
the vector emanating from this point. The ten most recent estimated target 
locations are the dots in and around the rectangle. The torpedo is shown 
as a triangle which represents the ensonification region of the torpedo's sonar. 
In the picture the torpedo is on an intercept course toward the target; the 
three most recently calculated intercept points are represented by "I's." 

The torpedo's ten most recent positions are shown as an almost-straight-line 
string of dots behind the torpedo. The alphanumeric information at the top 
and right side of the display is used by the operator to decide when to change 


torpedo course and when to turn on the torpedo's homing sonar. 


4.4.6 System Interactive Performance Evaluator. 


A system interactive performance evaluator (SIPE) was developed to 
aid in the analysis and design of a new area defense system for the Naval Sea 
Systems Command (NAVSEA 03). This work was done under a series of contracts, 


namely: 


@e N61339-73-C-0123, $62,920, May 1973 to July 1974 for the 
Naval Coastal Systems Center (NCSC) | | 

© N61339-74-C-0087, $71,178, March 1974 to March 1975 for NCSC 

@ N00024-75-C-4055, $112,526, November 1974 to July 1976 for NAVSEA 


The program monitor for this work was Mr. Bill Welsh, Code 03424 (telephone: 
202 - 692-1192). 7 7 | 


Two versions of SIPE were developed. One version was primarily used 
in the development of a system to defend against combat swimmer attack. The 
second version was developed for a large scale coastal defense system whose 
threat spectrum included surface craft, cruise missiles, and submarines. | 
Both versions allowed the input of detailed probability distributions whitch 


specified both the spatial and the temporal performances of individual system 
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Figure 7. Display Format During Execution of | 
| Interactive Control of Simulated Torpedo. 


elements (sensors, decision elements, and countermeasures/neutralization sub- 
systems). Once deployed, the simulator would Operate against targets which 
advanced along complex, input-specified trajectories. For instance, in the 
Swimmer case, not only the horizon plane trajectory of the swimmer could be 
specified, but also the swimmer's vertical trajectory. This made the swimmer 
visible to various sensors with different degrees of detectability. SIPE also 
allows the modification of sensor performance as the advancing threat passes 
through regions of clutter or other areas where sensor performance would be 


degraded. 


These versions of SIPE were used for three years to analyze area 
defense system designs. The SIPE simulation models conform basically to the 
Monte Carlo next-time-step and event-step configurations. The SIPE output 
consists of probability distributions which describe various critical events 
From the attack scenario, e.g., time and location of first detection, time and 
location of neutralization, relative frequency of threat penetration along 


given approach trajectory. 


Figure 8 shows detection and neutralization results on the display 
after simulating an attack by three swimmers against the Trident Support Site 
at Bangor, Washington. The swimmer tracks are shown in red. The sector 
coverages of two sonars are shown in yellow. Sensor degradation regions are 


shown as dotted contours through which swimmer trajectories 2 and 3 pass. 
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Figure 8. Results Shown on Display After Simulating a System Concept 
Against a Simultaneous Attack By All Three Trajectories. 


5.0 COST PROPOSAL 


5.1 © COST SCHEDULE 


All elements of our cost proposal are presented on a DD633-4 form. 
(See Exhibit 1.) 


5a2 LABOR RATES 


The proposed rates for all labor categories are weighted average 
expected rates, during the contract period, of personnel who will be assigned 


to the proposed study. 


5.3 OVERHEAD RATE 


Our overhead rate for 1978 was 111.8%. (See Exhibit 2.) We do not 
yet have an approved rate for 1978 because we have not yet been audited for 
1978. We anticipate that our rate during the term of the proposed contract 
will be 101%. All elements of overhead expense considered are in accordance 
with Section XV of ASPR. Because of the homogeneous nature of our present 
and planned business, we have found it advantageous to use only a single 
composite overhead rate applied to direct labor. We have no separate over- 


head for general and administrative expense. 


5.4 PER DIEM 


Overnight travel will be required in order to perform the required 
work. We propose that per diem be the same as allowed civil service employees 


on travel. 


5.5 CONTRACT TYPE 


A cost - plus - fixed-fee contract is desired. 
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EXHIBIT 1. 


DEPARTMENT OF DEFENSE 


CONTRACT PRICING PROPOSAL 
(RESEARCH AND DEVELOPMENT) 


Form Approved 


Budget Bureau No. 22-R1i00 


This form is for use when (i) submission of cost or pricing data (see ASPR 3-807.3) is 
‘required and (ii) substitution forthe DD Form 633 is authorized by the contracting officer. 


NAME OF OFFEROR 
Integrated Sciences Corporation 
) HOME OFFICE ADDRESS (Include ZIP Code) 


1640 Fifth Street 
Santa Monica, California 90401 


SUPPLIES AND/OR SERVICES TO BE FURNISHED 


Evaluating Effectiveness of Military 
Decision Support Systems 


[ DIVISION(S) AND LOCATION(S) WHERE WORK IS TO BE PERFORMED| TOTAL AMOUNT OF PROPOSAL | GOVT SOLICITATION NO, ; 
| Address above $ 90,339.00 N.A. 


DETAIL DESCRIPTION OF COST ELEMENTS 


[PAGE NO. NO. OF PAGES : 
| l 2 7 


| DIRECT MATERIAL (Itemize. on Exhibit A) |EST COST (3) weTOTAL ah 


‘ a. PURCHASED PARTS ® q . 200. 00 
Saese Ral LR 


c. OTHER - (I) RAW MATERIAL ited 


(2) YOUR ST ANDARO COMMERCIAL ITEMS 


(3) INTERDI VISIONAL TR ANSFERS (At Biker. than cost) 


Pino | TOTAL DIRECT MATERIAL 88 


| 2. MATERIAL OVERHEAD 3 (Rate % X $ base) 


| 3. DIRECT LABOR (Specify) | Gaal = ott 


| Chief Scientist . 


TOTAL DIRECT LABOR 
| 4. LABOR OVERHEAD (Specify department or cost centor)3 O.H. RATE EST COST (5) | 
alculated as a percentage of dj labor 1 36210 2,180.72 |: 


5. SPECIAL TESTING (Including field work at Government installations) — 


TOTAL SPECIAL TESTING _ PHIEEISIEINS 
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| 6. SPECIAL EQUIPMENT (If direct charge) {” 
A TRAVEL (ff direct charge) (Give detail? 
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| 8 CONSULTANTS (Identity - 
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| 7 EXHIBIT A - SUPPORTING SCHEDULE (Specify. If more ‘space is needed, use blank sheets) _ 
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HAVE THE DEPARTMENT OF DEFENSE, NATIONAL AERONAUTICS AND SPACE ADMINISTRATION, OR THE ATOMIC ENERGY COM. 
MISSION PERFORMED ANY REVIEW OF YOUR ACCOUNTS OR RECORDS IN CONNECTION WITH ANY OTHER GOVERNMENT PRIME 
CONTRACT OR SUBCONTRACT WITHIN THE PAST TWELVE MONTHS? . 


XX Yes [ ] NO If yes, identify below. 


NAME, AND ADDRESS OF REVIEWING OFFICE n ty 
Defense Contract Audit Agency, 95 
10.30 Ant non Ure 


- t' 


de ZIP Coda) 
S. LaCienega Bivd,, 


TELEPHONE NUMSBER/EXTENSION 


(213) 643-2629 


tl. WEILL YOU REQUIRE THE USE OF ANY GOVERNMENT PROPERTY IN THE PERFORMANCE OF THIS PROPOSED CONTRACT? 


[ ]} ves 


gil. DO YOU REQUIRE GOVERNMENT CONTRACT FINANCING TO PERFORM THIS PROPOSED CONTRACT? 


KX] No. if yes, identifv: [_}] apvance Payments [_]} PROGRESS PAYMENTS OR [_ | GUARANTEED LOANS 


[] ves 


ed NO If yes, identify on & separate page. 


IV. DO YOU NOW HOLD ANY CONTRACT (or, do you have any independently financed (IR & D) projects) FOR THE SAME OR SIMILAR WORK. 


CALLED FOR BY THIS PROPOSED CONTRACT? 


[] ves 


ky NO If yes, identify 


IV. DOES THIS COST SUMMARY CONFORM WITH THE COST PRINCIPLES SET FORTH IN ASPR, SECTION XV (See 3-807.2 (c) (2))? 
YES [ NO Ifno, explain on a separate page. . 


INSTRUCTIONS TO OFFERORS 


1. The purpose of this form is to provide a standard format by 
which the offeror submits to the Government a summary of in- 
_f|-curred and estimaced cost and attached supporting information) 
| suitable ror detailed review aud analysis. Prior to the award 
fof a contract resulting from this proposal the offeror Shall, 
under the conditions stated in ASPR 3-807.3, be required to 
i submit a Certificate of Current Cost or Pricing Data (see ASPR 
| 3-807.3(e) and 3-807.4). 


2. AS part of the specific information required by this form, the 
ft offeror must submit with this form, and clearly identify as such, 
f cost or pricing data (that is, data which is verifiable and fac- — 


i tual and otherwise as defined in ASPR 3-807.3(e)). In addition, | 


ihe must submit with this form any information reasonably requir- 
f ed to explain the offeror’s estimating process, including: 
a. the judgmental factors applied and the mathematical 
or other methods used in the estimate incinding thacea 
used in projecting from known da’ 


b. the contingencies used by © 
price. ? 
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3. When attachment of supporting cost or pricing data to this 
form is impracticable, the data will be specifically identified 
and described (with schedules as appropriate), and made 
available to the contracting officer or his representative upon 
request. 


4. The format for the ‘‘Cost Elements”’ is not intended as 
rigid requirements. These may be presented in diffe:ent 
format with the prior approval of the contracting officer if 


_ required for more effective and efficient presentation. In all 


other respects this form will be completed and submitted 
without change. 


5. By submission of this proposaLofferor, if selected for 
negotiation, grants to the contracting officer, or his author 
ized representative, the right to examrine, for the purpose of 
verifying the cost or pricing data submitted, those books, 
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EXHIBIT 2.. 


INTEGRATED SCIENCES CORPORATION 


Overhead Expenses for 1978 


Rent 

Office & Equipment Maintenance 
General Insurance 

Office Supplies 

Printing & Stationery 
Telephone | | . 
Postage 

Legal & Professional Expense 
Equipment Rental 

Depreciation 

Library Expense 

Corporate Car 

Marketing Travel & Misc. Expense 
Hiring Expense 

Secretarial Support 
Miscellaneous Expense 

Local Taxes & Fees 


Non Personne] 


Annual Leave & Holiday Pay 
Employee Bonus Plan 
Federal Payroll Taxes 
State Payroll Taxes 
Educational Benefits 
Employee Welfare ~ 
Professional Development 
Group Insurance 


Personnel 
Indirect Labor 
Marketing Labor 

Subtotal 
Estimated State Tax 

Total Overhead 


Direct Labor 


Dollars 


27,956. 88 
513.86 
1,811.00 


4,089.19 


1,085.34 
8,244.76 
779.32 
2,770.00 
4 824.8] 
7,014.21 
1,493.51] 
3,735.65 
1,430.60 
503.00 
251.73 
1,586.7) 
1,314.55 


69,405.12 


28,785.41 
2,106.20 
11,174.58 
2,585.47 
618.50 
490.23 
2,782.12 
13,222.36 


61,764.87 
38,994.56 
21,767.50 

191,932.05 
1,259.16 
193,191.2] 


172,840.17 
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5.6 ISC PERSONNEL TO CONTACT ABOUT THE PROPOSAL 


Proposal Area Name ‘Company Position | 


Cost Proposal David Walsh Treasurer 


Technical Content Antonio Leal Chief Scientist 


Mr. Walsh has the authority to negotiate and enter into a contract for the 


company's services. The above personnel can be reached at (213) 393-0257... 


5.7 FINANCIAL CAPABILITY 


Financial requirements for carrying out the proposed work will be 
satisfied with current ISC financial resources and progress payments received 
from monthly billings during the term of the contract. A balance sheet, 
profit and loss statement, and notes to the financial statements are presented 
in Exhibits 3, 4, and 5. 


5.8 RELATED CONTRACTS 


ISC has recently performed research on several large projects requiring 
skills similar to those needed for the proposed work. Synopses of some of © 


these contracts are given below. 


a. Design of Man-Machine Interfaces for the Navy's Operational | 
Decision Aiding Program (Contract N00012-75-C-0811) $344,800. This work is 


being done for the Office of Naval Research and includes the following: 
@ Simulation of Complex and Dynamic Systems on a Multi-Color 
Display 


e Development of algorithms for solving the defense penetra- 
tion problem for carrier-launched air strikes 


e Design of Advanced Man-Machine Interfaces for use by Naval 
Commanders to solve the defense penetration problem for 
carrier-launched air strikes 
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EXHIBIT 3 
INTEGRATED SCIENCES CORPORATION 


STATEMENT OF INCOME AND RETAINED EARNINGS 


December 31, 


through 


1973 


December 31, 1978 


1974 


°84 417 
°, 736 
580 


500) 


180 


1975 


1749 


(5484) 


5184 
3043 


$452,115 | 
415,318 


$ 36,797 


(3,735) 


$ 33,062 


8,227 
-0-— 


$ 24,835 | 


72,676 


$97,511 


__1,572 


5995939 


3934 


1976 


$666,466 © 
621,167 © 


$ 45,299 © 


77] 
(2364) (1,593) 


$ 43,706 


2293 
9,227 


1,266 . 
ea 


95,939 
$129,152 


: " 


$129,152 


1977 


$391,021 
396,792 


($5,771) 


1936 


(1234) 702 


($ 5,069) 


(1054) 


200 (854) 


1,267 
($ 5,482) 


129,152 
$123,670 


2,738 


$120,932 


1978 


$427,925 
412,364 


$ 14,96] 


(444) 
9 1455) 7 


560 
(1004) 


2642 


1307 3,949 


1,267 
9,301 | 


120,932 
9130,233 


1, 356 


$128,877 


“- 


Current Asse’ 


eS os 


$ 19,578 


EXHIBIT 4 


INTEGRATED SCIENCES CORPORATION 
BALANCE SHEET 


December 31, 1973 through December 31, 1978 


ome Le 


> 28,535 
30,983 
36,089 
77,242 

22 

__-0- 

$172,871 


82,225 


(19,108) 
2 635117 


$ 1,454 


$237,442 


10,091 


72,676 


$100,835 


$237, 4h2 


see IO os 


$ 34,383 
~Q- 
55,386 
62,791 

D, 150 
-~Q- 
9157,710 


96,042 


(36,235) 


> 59,807 


$1,454 


$218,971 


~—6$ «18,068 


11,793 
95,939 


$125,800 
$218,971 


_1976 

$ 13,088 
32,082 
56,42] 
65,086 
2,043 
-0- 

$168,720 


139,116 


(54,731) 


$ 84,385 


$ 4,698 


$257 , 803 


$ -0- © 
17,873 
17,015 
2,014 
19,560 
32,820 
pre by, 

9 95,221 


$ ~0- 


> 95,221 


$ 18,068 
15,362 


129,152 


$162,582 


$257,803 


1977 


9 9,775 
23,696 
18,733 
54,040 
17,949 
4,843 


| $129,036 - 


147,348 


(77,710) 


$ 69,638 


93,762 
$202,436 


$_ -0- 


$ 48,074 


$ 18,068 
15,362 


120,932 


$154, 362 


1978 
$ 15,676 
8,222 
69,014 
31,851 
38,00] 
-0- 
$162, 764 


147,348 
(101,713) 


aS 45,635 


$__ 3,762 
$212,161 


S 8 
18,541 | 
-0- 

1,183 
63,785 
23,341 

2,996 

§ 49,854 


$ -0- 
§ 49,854 


$ 18,068 
15,362 
128,877 


$162, 307 


$212,161 


EXHIBIT 5 


INTEGRATED SCIENCES CORPORATION 
NOTES TO FINANCIAL STATEMENTS 


The company has a five-year lease on its office space at the rate of 
$3,530 per month. The lease expires in June 1981. 


In February 1972, the Board of Directors of the Corporation approved a 
qualified stock option plan under which 2,000 shares of the Corporation's 
common stock are available for grant options to key employees to purchase 
stock at no less than market value at date of grant. There are currently 
no outstanding options to purchase shares. 
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@ Design and Monitoring of Experiments 


e Statistical Analysis of Experimental Results 


b. (Contract MDA903-78-C-2012) 


$66,984. This work is being done for the Army Research Institute and includes 


Tactical On-Line Maneuver Modelin 


the fol lowing: 


e@ Development of a ''look ahead'' battlefield model to take 
advantage of user oriented interactive graphic displays. 
The model provides straightforward user techniques for 
showing both friend and enemy unit types, unit levels, 
basic movement characteristics and engagement/firing actions 
on an automated display system. 


e@ Calculation and display of battlefield mobility character- 
istics for different unit types. 


e@ Calculation and display of contours representing the 
potential future locations of a unit. 


c. Interactive Defense Penetration Analysis for Submarine-Launched 
Cruise Missiles (Contract NO0O140-78-C-6141) $55,000. This work was done for 
the Naval Underwater Systems Center, Newport, Rhode Island. It included the 
following: | 
@e Definition of the overwater cruise missile control problem 
Generation of missile trajectory optimization models 


e Development of interactive techniques for using computer 
- graphics. and the trajectory optimization models for solving — 
the trajectory generation problem for a variety of launching 

conditions. : 


d. Interactive Modeling of Tactical Events (Contract DAHC1 9-77-C- 


0022) $60,563. This work was done for the Army Research Institute and 


includes the following: 


e Design of a man-machine interface, including display formats 
and control mechanisms for entering and editing enemy tacti- 
cal hypotheses. Development of operating interactive 
graphics software to implement the design. 


e Development of simulation models of mobility, detection, 
classification, localization, and false alarm performance 
of intelligence gathering activities against the classes of 
enemy targets used in structuring tactical hypotheses. 
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e Design of the man-machine interface required ae compressed 
time replay of enemy tactical hypotheses against simulation. 
of intelligence gathering activities so that a decision 
maker can compare his hypotheses against the intelligence 
data and decide which hypothesis is most consistent with 
the data. Development of operating interactive graphics 
software to implement the design. 


e. Scientific and Technical Services to Conduct a Baseline Study of 
Effects of Color on the Man-Machine Interface Performance of Underwater Fire | 
Control Systems (Contract N0O0140-76-C-6272) $39,214. This work was performed 
for the Naval Underwater Systems Center, Newport, Rhode Island. The work 
includes the following: | 
e Design of alternative man-machine interfaces of a weapon 
order generator on a submarine fire control system 


e Test and evaluation of interfaces designed 


f. Systems Analysis and Human Factors Studies (Contract N00123-75- 
C- 1202) $926,490. This work was performed for several programs at the Naval 


Undersea Center, San Diego. The work includes the following: 
e Application of pattern recognition eoCnnVanes to design of 
a bionic sonar 
Human engineering 
Design of an advanced deep ocean sonar system 
Requirements Analysis 
Cost Effectiveness Evaluations 
Operations Analysis 


Test and Evaluation Procedures for Research Studies 


Personnel Subsystem Development 


-g. Human Engineering Design Study to Support Defense Against Swimmer 
Attack (Contract N00123-73-C-0289) $84,533. This work was performed for the 


Naval Air Station, Point Mugu, California. The work included: 


@ Man-machine interface design of a console used by an 
operator of a swimmer defense system 


e Test and evaluation of interfaces designed 
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ABSTRACT 


Decision Analysis is a methodology of selecting a course of action 
based on numerical encoding of an individual's value-system. The paper dis- 
cusses the possibility of empirically demonstrating the merit of submitting to 
such methodology, as well as the feasibility of testing the fidelity of value- 

' quantifying methods. 

We dispute the notion that the aim of utility elicitation is to encode a 
subjective value-system which in itself escapes external control or exami- 
nation. We argue that the very value-system that utility functions attempt 
to capture is an encoded version of some empirically observable experience. 
The merit of any value-elicitation technique should therefore be judged by 
its ability to closely reflect the essentials of this experience. Correspond- 
ingly, we propose to test the effectiveness of Decision Analysis procedures 
by their ability to improve the performance of laboratory eined subjects 
operating in a controlled simulated environment. The simulated environ- 
ment presents decision situations with their associated ene in accor~ 
dance with some utility funchien:, thus, guaranteeing consistency and 
| uniformity. Score improvements recorded after subjects submit to 
Decision Analysis routines should be used to improve vate elicitation 
techniques, identify areas where systematic errors are likely to occur, 


and to assess the cost-benefit tradeoffs of Decision Analysis practice. _ 
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I, INTRODUCTION 


This paper was motivated by a discussion that took niseé an a recent 
workshop on "Subjective Optimality" conducted at the Cees of Michigan in 
July 1972. After hearing several reports on the application of Decision 
Analysis in various problem areas a question was raised whether there exist 
some means of measuning the benefit incurred by organizations who solicited 
the assistance of decision analysts in their specific problem areas. in the 
discussion that followed an argument was made that any meaningful claim for 
success should (at least in principle) be submitted to empirical test. Some 
Decision Analysis practitioners, on the other hand, claimd that their success 
does not lie with ecernaity observable entities such as profitability or pros- 
perity but rather with implementing decisions which are in conformity with 
the decision maker's subjective value judgment, and so escapes empirical 
scrutiny. The present paper aims at further analysis of this dilemma leading 
to a constructive method of validating ane of the claims made by decision 
analysts. 

The difficulty in submitting Decision Analysis (DA) to empirical test 
can best be presented after a distinction is made between DA's methodology, 
jovieal basis, and sipieieal claims. We shall start with a short description 
of the activity taking place when a decision analyst is called upon to aid ina 


(1) 


decision-making process. 
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Activity 
od. Problem formulation — Identifying crucial variables, and separating 
them into control variables, state variables and consequences. 


2. Elicitation of subjective probabilities connecting the states of nature 


and the consequences (outcomes). 


ce Flicitation of the relative value (utility) of the outcomes to the decision 
maker. 
4, Calculating that action which among all available alternatives maximizes 


the expected subjective utility. 

Though many iterations of these sequences in various orders may 
sometimes by necessary, at any given time the decision analyst will be found 
involved with one of these four activities. 

Direct vs. Indirect Benefits — It is not difficult to scan the list of 
activities above wal reason how each may lead to beneficial results. Any 
moment of thought spent by the decision maker on analyzing cause-effect rela- 
| tions, listing alternatives, facing eoneeauesece, recounting past experiences, 
may sharpen his visibility and enhance his chances of making better decisions. 
These effects, however, we should call indirect benefits as they do not 
directly relate to the particular methods employed by DA but would accom - 

_ pany any arbitrary procedure which induces the decision maker to focus atten- 
tion toward hebiey understanding of his problem. Indirect effect of this ‘aii 
will not be part of our considerations as even the most conservative aon 
analysts claim to offer acne service than merely serving as sounding boards 


to frustrated executives. 


a 4a 


Another indirect effect not included in our evaluation is the benefit 
drawn from people's peculiar respect for written procedures. This effect 
was found to reduce tension among individuals as the final decision (usually 
submitted in the form of computer print out) seems detached ice interper- 
sonal conflicts. The written procedure was also shown to ee anxiety 
among decision makers as it provides a potential scapegoat in case of blun- 
ders. 

Our objective is to evaluate the direct benefits of the unique misieds 
and procedures used by DA as opposed to some other rigid methodologies. 

Logical Foundations of DA — The logical basis for DA methodology is 
provided by the celebrated axiomatic treatment of Van Newacnn and 


(2) 


Morgenstern (VNM). The latter states that if an individual's preferences 
among lotteries obey a certain set of axioms then his preferences can be 
represented (or encoded) by a real function on the outcomes, called utility, 
in such a way that lottery A will be preferred to lottery B if and only if A 
results in a higher expected utility than B. When the decision problem con- 
wins uncertain events whose likelihood cannot be measured by relative fre-_ 
quency measurement then additional tein must be supplied concerning the 
individual's ae of reacting to Gcertintics relative ke his reactions toward 
repetitive events with well established probabilities, o When these axioms 
are added to VNM axioms it aan then be shown that the individual's seus . 
toward uncertain events can be encoded by a "subjective" probability function 


and all his preferences can be determined by the "subjective" expected 


utility. 
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All this is well known and was presented here merely for contrasting the 
iene foundations of DA with its empirical claims. Decision analysts do not 
claim that individuals do behave in accordance with VNM axioms, for if they 
do they would not solicit analysts’ assistance. They claim however that most 
eaigidaats would like to (and therefore perhaps ought to) behave that ean 
This last statement is an announcement of an empirical fact which can be 
submitted to and has indeed been confirmed by empirical tests. The function 
which the decision analyst undertakes is to assist an individual decide "the 
way he really likes to act". Todo that, the decision analyst exposes his subject 
to eomiodiaal decision situations and from his responses synthesizes two 
oe of functions: subjective probability, and utility. 

The degree at which the decisiGd analyst poeeeeds in eliciting these 
functions, and their relations to the entire preference system are the crucial 
elements in the problem of DA evaluation. While ie elicitation of subjective 
probability can be (and has been) submitted to empirical tests, ae the fidelity 
of encoding one's value judgment in the form of a utility function has defied 
: macasurements. In fact, some decision analysts argue that such a measure- 
ment is an impossibility. 

The argument goes as follows: Testing the fidelity of our value elicita- 
tion method can only make sense if we find another means of measuring inter- 
nal value systems. But internal value systems have only one form of external 
manifestation: through preference value Judgments. Now, our method of 


value-elicitation is based on exactly this manifestation, therefore the utility 
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we come up with is, by definition, the subject's value system and no test in 
the world can refute or support this claim. 

This argument may have settled the entire problem, was it not for an 
uneasy feeling that overcomes every empiricist hearing claims which defy 
empirical verification. Especially when he is asked to pay fat fees for imple-_ 
menting procedures based on such claims. His immediate response ina case 
like this would be either to dismiss the claims as metaphysical, or attempt 
and identify their meaningful and testable comosneais. He may, for instance, 
ask waseli: Would I be willing to pay a fee to a witch-doctor claiming to 
precisely measure the color of my intelligence? Here is an entity which is 
defined solely by the claimer's elicitation method; is our value-system of the 
Same semantic level? I souldars care less for the color of my intelligence 
but I sure would like to know my scale of gebese, It is exactly this differ- 


ence that exposes the empirical basis underlying value-elicitation. 


Ii. EMPIRICAL FOUNDATIONS OF VALUE ELICITATION 

We would like to identify those empirical facts and claims which provide 
the necessary eee between axiomatic decision theory (which in isolation 
remains purely tautological) and prescriptive decision theory as apolied to 
real-life human decision making. For this purpose we first address our- 
selves to the question: What aspects of human behavior give rise to the notion 
that human decision making ought to be governed by a "value system"? 

Let us consider the set C of all data of choice behavior produced by an 


individual over a long observation period in a certain decision-making envi- 


ronment. C may contain the individual's responses to a large number of — 
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questions posed to him by a passive (non-advising) experimenter as well as 
choice preferences reflected by ii actions. In general C will consist of a 
set P of preferences among specific choice alternatives and a set U of gen- 
eral principles about the structure of thesetonetercncse, Statements which > 
belong to U are distinguished from those of P by the presence of universal or 
existential quantifiers. For example, the statement expressing He subject's 
acceptance of transitivity will belong to U as it expresses a universal prin- 
ciple compatible with all objects of choice, while the statement "I prefer lot- 
tery B" will belong to P. 


The first empirical truth that we need to consider is: When allowed to 


operate without advice most individuals generate logically inconsistent 


C's. en?) This truth is substantiated by the very fact that decision analysts 


are often called upon to offer advice. Had C been free of contradictions the | 
best advice the decision analyst can come up with would be to Bet his client 
which alternative he prefers eres and aeael his full support. That would 
amount to a sheer duplication of C, 

The contradictions contained re C are of several types. They can be in 
a form of two or more incompatible preferences like 'A is preferred to B! | 
and 'B is preferred to A', both appearing in C, There can be a conflict 
peteen a universal principle and a set of particular preferences, or evena 
conflict between two universal principles. We usually attribute these contra- 
dictions to limitations in human computational powers. The logical path 
between the universal peeinie: in U and some of the decision situations in 


P may contain very long chains of deductive reasoning and arithmetical steps 
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of which we Siaiy cannot keep track. One of the recognized merits of DA 
procedures is that this computational burden is substantially reduced via the 
employment of numerical codes which can be manipulated in a mechanical 
way. 

But before one can employ numerical codes one must decide what he 
wants to code, especially when the object to be coded is imbedded in a logi- 
cally inconsistent space. Since any numerical decision procedure is bound 
to be self-consistent we always expect to find deviations between some state- 
ments in C and the corresponding statements as derived from numerical pro- 
cedures. Various decision procedures may differ, however, in the set of 
statements which they deviate rom C. The latter depends on the particular 
choice of basis employed by the procedure. 

A basis B is a subset of statements in C which is chosen for numerical 
encoding. Decision Analysis, for example, selects as a basis the set of uni-~ 
versal principles from U known as the Von Newmann Riernenstena (VNM) 
axioms, and the set of particular preferences from P used by the analyst to 
elicit subjective probabilities and utilities. For the Suecessnil employment — 
of this basis a pagal wacititaes fact must be true: Most individuals admit 
VNM axioms in their C records. This fact has indeed been supported euseu: 
imentally. 

Once encoded, the chosen basis can be used to generate a self- 
consistent set C(B) (called the logical extension of B) of preferences and 
principles (theorems) which agrees with most, but disagrees with some state - 


ments inC. Adherence to a decision procedure implies that the decision 
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maker always takes the action recommended in Ca even when it conflicts 
| with some statements he has made or may later admit in C. 

But the choice of basis is quite arbitrary, any complete and self- 
consistent subset of C might serve ae a basis for a numerical decision mak- 
ing procedure. What then are the aspects of C that make the basis ee 
by DA better than any other choice? The following example will demonstrate 
the guidelines we ought to adopt for proper basis selection. @encics: a set 
C which among many others also contains the following statements: (1) VNM 
axioms, (2) Indifference to a trip to the Bahamas or $1000 in cash, (3) 
Indifference to a trip to the Bahamas or $1200 in cash, (4) $1200 is pre- 
ferred to $1000. Obviously C contains a contradiction, but one which we are 
very likely to make when asked to compare outcomes outside our field of 
familiarity. Consider now the ramifications of choosing two bases: 


B= (1), (2) (4) B, = (1), (2), (3) 


‘iiieini toa procedure employing B, would mean that our subject should 
give up statement (3) and eubecnibe instead to: (3') "A trip to the Bahamas 
is inferior to $1200 in cash." On the other hand, a procedure employing B 9 
would force him to give up (4) and replace it with (4') "Indifference to $1200 
or $1000, " 
Intuitively we all feel that B is more representative of the record C 

than B 3° Few of us will be willing to subseribe to (4!) while the acceptance 
of (31) is quite harmless in view of the fact that we are usually not too profi- 
cient in assessing the fair market prices of vacation trips. It is not too hard © 


to construct more elaborate examples of this sort to demonstrate how 
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| inpeOReE Gngice of basis can cause many seemingly minor inconsistencies in 
C to be combined cumulatively and yield results which stand in complete con- 
tradiction to common sense (Plato's dialogues are classical examples of such 
technique). At the same time, there is nothing in C which dictates what con- 
stitutes a good basis. What then is the nature of the "value system" that deci- 
sion analysts claim to capture while eliciting utilities? 

As the previous example eS pecneteetes. the ingredient missing from the 
raw data of choice enayisE (C) is an indication of the individual's strength of 
sony ess in each of the choice statements. In the example above statement 
(4) will invariably be expressed with more determination than either (2) or (3). 
The reason we tend to prefer B, over By is that CB.) 1s in cnetieas witha 
statement of high strength of conviction while CAB 1? violates a statement we 
weren't ioe sure of to begin With. 

If it were possible to obtain reliable measurements of the "strength of 
conviction", we could saxhaps devise meaningful ways of evaluating decision 
analytical procedures. We would first define individual's value system V to 
‘be the set C of behavioral choice statements coupled with a set of real numbers 
(ranging, say, from 0 to 1) indicating the strength of conviction assigned to 
each of the statements in C, thus, V C C x Re. A decision procedure aulie 
ing in choice system ae will be evaluated by the degree of its conformity with 
the value systems of the subject. The degree of conformity should be some 


decreasing function of the sum of the statements in which C conflicts with C. 


weighted by the strength of conviction of these statements. 
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The claim for success of decision analysts would then be phrased as 
follows: 
We have found a method of selecting among the various members of 


C a subset B of statements which is both complete and consistent and 


whose logical extension C (B) stands in close proximity to the value 
system V. 


The empirical verification of each aie wou entail comparing C (B) and V 
for a large number of individual subjects. 

In eeeence several decision analysts have come very close to this 
formulation. They claim that through lengthy in depth sigewevions ies are 
able to single out those variables and situations with which their clients feel 
most confortable and confident. Correspondingly, by way of verification 
they propose measuring clients satisfaction after implementing their deci- 
sion procedures. Surely, a cine could not stay satisfied for a long 
period of time if some of the decisions recommended to him on a computer 
print-out stood in direct conflict with his strongest convictions. 

The drawback in relying on clients' satisfaction as a measure of 
success is twofold: first, some ramifications of a decision procedure may not 
be known for a long period of time until uncovered by a peculiar choice situa- 
tion. For instance, in the example cited above we may be unaware that the | 
procedure to which we subscribe implies indifference beewesn $1000 and $1200, 
That fact might not reveal itself until ie are actually faced with a choice situ- 


ation involving sums of money in the range $1000 to $1200. 
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Second, and much more fundamental, sutisfaction is a measure of 
conformity to one's value system after analysis. It provides no indication to 
the degree of value-bias induced by the very practice of the procedure under 
test. Indeed many decision analysts pride themselves on the fact that some 
of their clients undergo complete restructuring of their value-systems asa 
result of the analysis. The vague becomes clear, what seemed crucial before 
may become less important, and strong convictions may turn out to be sheer 
wishful thinking. What value systems were these analysts trying to salina 
The one before or the one after analysis? If it is satisfaction that we are try- 
ing to create can't we do better by subjecting the decision maker to some 

sort of a self-hypnosis program (with the help of drugs perhaps) as a result 
of which he emerges satisfied with almost any decision he makes. 

These questions lead to the realization that although satisfaction is a 
prerequisite for any successful decision procedure it cannot be regarded as a 
goal in its own right. We must define another observable manifestation of 
our value system aside from the collection of choices that brings about satis-_ 
faction. We can of course insist on taking the data V © C x Re before analy- 
sis to represent the decision-making true value-system. But our objective 
can be served more effectively if we bypass this tedious exercise and reexam- 


ine the reason we wish to consult our value system in the first place. 


II. TESTING ELICITATION OF VALUES 
We constantly struggle to make decisions which conform to our value 
system. To explain this phenomena from a pragmatic viewpoint, we must aban- 


don the notion that value is an internal mental state isolated from our external 
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environment. The reason we consult our value system is that in it we hope 
to find a condensed version of our past experience to guide our future actions. 
The value system that we defined earlier as the set V is in itself a product of 
a mental encoding procedure whose objective is to capture the essentials of 
our environmental cultural and intellectual Sctiviies. If, for instance, we 
prefer to make decisions in accordance with the culate of transitivity it is 
not because we were born with an innate favor toward transitive relations but 
because we simply fail to recall from our experience any set of three pairs 
(A, B), (B, C) and (C,A) in which the first member in every pair produced 
more reward than the second. (A, B and C represent states of the world.) 

. Preferences among epaetic situations and their associated strengths of con- 
viction are similarly aeauied via some process of averaging over one's past 
exposure to the situations and the reinforcements they carry, or with some 
aspects of those situations. 

With such a behaviorist picture of value formation in mind we might 
ask: Why evaluate Decision Analysis on the basis of its fidelity in eliciting | | 
one's value system, why not measure its fidelity in coding the very experi- 
ence which brought about that value system. 

In laboratory conditions we can synthesize an artificial environment of 
choice-reward consistent witha well specified utility function. A group of 7 
subjects is then instructed to "learn to aunave” in this environment, namely 
to eyaite decisions which maximize their reward. After a certain training 
period, operating scores are taken and the group is placed under the ee 


sion of a decision analyst who attempts to elicit their utility function (with © 
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respect to said environment) and subsequently assists them in employing the 
ulility function toward bisacediien tutte decisions, The operating scores afew 
aunulysis are then compared with those tuken before unulysis. In addition the 
utility sieuea by the analyst is compared to the utility function underlying 
the environment and systematic errors are noted. 

Two objections may be raised against using this test for measuring DA 
effectiveness: The first is based on the fact that the differences between the 
seneesine utility and the elicited utility reflect | sum of two errors; learning 
error and elicitation error. The former affects coding the environment 
into the subject's internal value-system and the latter accompanies coding 
their value-system into a numerical utility function. It is argued that Dicision 
Analysis should not be held responsible for subjects learning deficiencies but 
solely for elicitation.errors and since the two are empirically inseparable 
the proposed test has no bearing on the effectiveness of Decision Analysis. 

Our answer to this objection is that the two errors are at least partially 
separable. The systematic component of the learning error should be observ- 
able in the pre-analysis scoring records and so can be subtracted from the | 
total systematic error. The non-systematic components may ba wasted out . 
by averaging over a large non-interactive group. 

The second objection refers to extraneous atieets caused by the 
laboratory set up. In real life situations, the argument goes, the decision 
maker is free to expose his entire value system while in the proposed test he 
is asked to respond to a smail subset of his system, that created by the labo-_ 


ratory environment. Since no one can completely divorce himself from past 


a ha 


values, experiences and aspirations we should expect much higher elicitation 
error in the laboratory than can actually be achieved in real life. 

Our answer to this challenge is that in most decision situations of any 
importance we are not at all interested in eliciting the entirety of the decision | 
maker's personal values. On the contrary, we are primarily interested in 
his = system with respect to a narrow cultural or organizational one 


(6) 


text. A public official, for example, is expected to act not in accordance 
with his personal value system whichis sean ce over the years, but rather 
his assessment of the public value system at a particular time with respect. 
to a specified issue. A corporate executive is expected to act in harmony 
with the risk Boley of his corporation, even when it conflicts with his own 
temperament. We, therefore, are not interested in one's total experience 
but rather in one's ability to discriminate and select those experiences which 
are relevant to his duty. Accordingly, we should be interested in measuring 
the ability of DA to elicit one's evoked or learned values rather than one's 
intrinsic value-system. The set up in the proposed test is designed to match 
this objective; the subjects will not be asked to state thets own personal pref- 
erences but rather the choices that ‘hey believe would result in a maximum | 
return in the artificial environment whieh they studied. 
The test procedure we described is similar in several respects to the 

tests we normally use to calibrate the elicitation of subjective probability. se 
In the latter case we record a set of personal peierenees related to the | 


occurrence of an uncertain event whose probability function we know exactly 


and from this record we construct a subjective probability curve. Having an 
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acon calibration device allows the evaluation of various elicitation 
techniques. We can, for instance, measure the effect of various question 
phrasing, audio visual devices, group interaction, etc. on the accuracy of the 
elicited probability curves. The systematic ene found in such tests‘® may 
be used to compensate later results in cases where the probabilities are 
unknown or untestable. Our finding an external calibration for values equips 
the encoding of utility with identical poseibiniies. | 
In case our test reveals a panied systematic error in value encoding 
it stands to reason that appropriate compensations be affected before deci- 
sions are enacted. Such value-errors have long been recognized by moral 
philosophers. We are often warned against underrating virtues such as Health | 
and Freedom whose "true value" can only be appreciated in their absence. | 
We are also reminded of our endenety to overemphasize factors which play a 
role in our most recent experience at the expence of those experienced in the 
past. While the sseeeenncats proposed in this paper may lead the way toward 
quantifying these and similar effects they A186 present Decision Analysis with 
a tough social challenge; stating what the subject's utility ought to be strips 
VALUE from the ‘sacred power it now possesses, a power whom all try to 
please and none dares aicocons Whercas people are generally willing to 
accept eatibration of their subjects probabilities, how many of us are will- 


ing to submit to value calibration? 
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A Framework for Processing Value Judgments 


JUDEA PEARL, 


Abstract—Traditional decision-analytic practice emphasizes the 
distinction between’ probability assessments and value (or utility) 
judgments. Whereas techniques for elicitation and integration of sub- 
jective probabilities often can be submitted to empirical tests of validity, 
the fidelity of encoding value judgments has so far defied measurements. 
A unified approach to the treatment of the two types of judgments ‘is 
presented; value judgments are interpreted as conditional probability 
statements. Such formulation leads to rational methodologies and pro- 
cedures for solving the following tasks: 1) empirical validation and 


refinement of value judgments and 2) aggregating value judgments . 


obtained from a panel of experts. 


]. INTRODUCTION 


; HILOSOPHERS have long separated between two 
Pring: of human knowledge, informative (or positive) 
knowledge and normative (or value) knowledge. In the 
first category philosophers grouped the physical and socio- 
economical sciences as well as our intuitive knowledge 
about objects, colors, geometrical shapes, etc. In the second 
category, which often enjoyed a more elevated status, we 
find the studies of ethics and religion. 

Strangely, this philosophical tradition has retained its 
influence upon the thinking and practice governing the 
emerging technology of decision analysis (DA). From its 
early development in the 1940's until today, DA has pre- 
scribed separate interpretations and treatments to prob- 
abilistic and value judgments. Whereas the former is 
regarded as a formalization of experiential knowledge, the 
latter is treated as a subjective and capricious entity which 
aside from the requirement of satisfying transitivity may 
assume any structure whatsoever. | 
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This distinction has resulted in a technological disparity 
between the methodologies available for processing prob- 
abilistic and value judgments. Whereas the former enjoys 
the benefits of Bayesian analysis, no parallel formulation is 
available for processing value judgments. The disparity is 
most clearly reflected in the absence of rational provedures 
for refining, updating, calibrating, and aggregating value 
judgments. While judgmental probabilities can be modified 
by experts, updated by empirical observations, and rectified 
by calibration, none of those techniques seems to apply to — 
value judgments. Value preferences seem to remain im- 
mune from analytical or empirical scrutiny, a thing to oP 
hold and not question, satisfy but not alter. 

Consider, for example, the problem of calibration. 
Calibration procedures are used to bring probabilistic 
codings given by an individual closer to the truth frequen- 
cies of previous judgments [1], [2]. These procedures are 
used to remove systematic biases from both the individual’s 
own perception of uncertainty and biases inherent in the 
elicitation method. Thus, if a probability assessor claims 
that his bread has fallen on the buttered side 90 percent 
of the time, we can use this overstatement to attempt to 
calibrate his prediction that his next car stands a 90 percent 
chance of being a lemon. However, if that same “expert” 
claims that he prefers arsenic to hot chocolate, we have no 
decision-analytic procedure of tempering this preference. 
It must be taken at face value and fed into an optimization 
procedure, which would produce an action conforming to 
almost any declaration depicting the expert’s desires, 
however careless. a 

Next consider the problem of aggregation. When a panel 
of experts provides conflicting probability assessments, 
Bayesian techniques are available [3] for combining these 
assessments into a single probability structure reflecting the 
group overall knowledge. These techniques treat the experts 
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as noisy information sources and incorporate the decision- 
maker’s judgment of their reliability into the aggregation 
rule. No parallel techniques are available for aggregating 
value statements. It is strange that Bayesian methodology 
can resolve experts’ disagreements on the failure probability 
of a nuclear reactor but could not handle disagreements 
concerning the social value of that reactor. 

The purpose of this paper is to explore a paradigm of 


value semantics which may facilitate extensions of Bayesian - 


decision-analytic techniques to the processing of value 
judgments and help rectify the imbalance now prevailing 
in our conception of the informative and normative modes 
of human knowledge. 


II. A PARADIGM FOR THE SEMANTICS OF 
VALUE JUDGMENTS 


The paradigm advanced in this paper asserts that value 
judgments and probability statements are one and the same 
thing. Both are numerical codes of experiential data, both 
are constructed by the same mental procedures, and both 
are utilized to guide future actions. 

If probability quantifies a person’s belief in the eventuality 


of a certain situation, then utility quantifies one’s estimate 


of the benefit he might eventually draw from a certain 
situation. It is reasonable, therefore, to regard value judg- 
ments as conditional probability (or conditional expecta- 
tion) statements. For example, the statement “I prefer 
outcome A to outcome B” may be interpreted to mean “I 
estimate tne probability of reaching a certain state of satis- 
faction conditioned upon A to be higher than that con- 
_ ditioned upon B,” or “I estimate the total benefit achievable 

as a result of realizing A to be higher than that achievable 
after realizing B.” In either interpretation the entity esti- 
mated by the value statement is the strength of causal con- 
nection between two eventualities; one being the outcome 
A or B, the second being a person’s achieving a certain 
level of satisfaction. In the framework of Bayesian analysis, 
though, the quantification of causal relations among events 
is usually formulated through conditional probability (or 
conditional expectation) assertions. To remain consistent 
with this formulation it behooves us to regard each value 
statement as a parsimonious form of conditional probability 
assertions. It should therefore be treated as an empirical 
statement about “matters of fact’? and should acquire a 
status equal to that of any probabilistic assertion which 
summarizes observational data. | | 

To complete the unification of value judgments with 
probability statements we need to clarify the concept of 
“personal satisfaction” or “total benefit.” In most people’s 
eyes satisfaction may appear in a variety of shades and 
colors not all of which seem commensurable. For example, 
the satisfaction one receives from dining in a certain 
restaurant seems noncommensurable with that induced 
upon hearing encouraging political news, or with the fear 
of adding so many extra calories to one’s diet. The unlike 


character of these kinds of satisfaction, however, seems to 


be derived from people’s tendency to attribute to satisfac- 
tion the character of the anticipatory procedures invoked 
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for estimating its future likelihood or future magnitude. 
The knowledge that added weight stands tn causal connec- 
tion to overeating, for example, serves an anticipatory func- 
tion of estimating the degree of future discomfort resulting 
from a certain meal. The nature of the discomfort itself, 
though, is fairly commensurable with the nature of the 
gratification one receives during the meal. Both involve 
biological sensations of pleasure and pain. 

Neurophysiology has found that many mammals’ brains 
(including humans) contain a so-called “‘pleasure center,” 
stimulation of which encourages that individual to seek 
further stimulation there. In rats, a conditioned reflex can 
be erected on this basis; in humans, the overt effect was that 
the patient asked the doctor to “Do that again, please.” 
From this low level of neural sensation, therefore, it is 
reasonable to assume that only two parameters suffice to 
characterize the force which drives an organism’s behavior: 
intensity and time duration. Formally we can express this 
assumption by stating that at any time an organism is 
attempting to maximize the expected value of a satisfaction 
integral : | 

oO 
S = { s(t) dt (1) 
t=to 

where s(t) measures the intensity of the pleasure sensation 
at time ¢ (a negative s should reflect painful sensations). 
Toda [4] used an identical formulation which he named an 
adaptance integral. S is a concept comparable to utility 
in the sense that, like utility, S is unidimensional and 
unique up to a linear transformation. However, unlike 
utility, S is not an attribute of a decision outcome, neither 
is it subjective in nature. It measures an actual neural 
activity of an organism undergoing a specific life history. 

To connect S with the usual concept of utility, we assume 
that the organism configures its experiential knowledge in a 
form which permits him to calculate (or estimate) the func- 
tion E(S | 0), where E stands for the expected value opera- 
tion and @ is a description of a state of the world. We then 
say that the utility of the state @ is given by ~ 


V(0) = E(S| 9). (2) 


- Clearly, the mental calculation of E(S | 0) is subject to - 


many biases and inconsistencies inherent In humans’ com- 
puting machinery. The main deficiency lies in man’s ability 
to organize past experience in a data structure that would 
permit him to calculate the probabilistic connection between 
S and 0 (for all levels of S and all states 0). A second Ihmita- 
tion lies in man’s capacity to compute E(S [ 0) on the basis of 
such probabilistic information. With these deficiencies in 
mind, our main assumptions at this point are 1) that an 
individual would attempt to harness his computational 
resources, however limited, to facilitate an approximate — 
calculation of E(S | 0), and 2) that once a crude estimate 
of E(S| 6) is obtained, it is this estimate which governs 
individual choice behavior. | | 

In our normal speech, the variable S does not appear 
explicitly as an argument of utility. We talk about utility of 
lotteries, outcomes or commodity bundles, tacitly implying 
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that the latter are only vehicles serving to achieve or estimate 
the total future benefit S to be experienced by our pleasure 
and pain centers.! 

The reason that the role of the senses in affecting value 
judgments has almost completely disappeared from our 
everyday language is several fold. First, the argument S 
remains constant in all value judgments issued by an indi- 
vidual at time fg, and so, it is redundant and may be dropped 
from a normal discourse. Second, the neural activity re- 
presented by S is so logically remote and perceptually un- 
certain that it rarely participates explicitly in the calculation 
of E(S | @). Chess masters, for example, concern themselves 
primarily with heuristic concepts such as mobility, tempo, 
center-control, etc., rarely mentioning the ultimate goal of 
the game: checkmate. In much the same way our cognition 
system is equipped with heuristic concepts such as the 
pursuits of social virtues, economical wealth, or political 
power which serve to guide our choices toward an inter- 
mediate milestone. These concepts do not in themselves 
guarantee a realization of high S but where chosen by our 
culture as useful computational tools in the estimation of 
E(S | 6), by virtue of their tight correlation wate future 
pleasure/pain balance. 

Formal models for the evolution and utilization of such 
value concepts were developed in the literature on artificial 
intelligence [6], [7], and will be used later in Section III. 


The main point to notice here is that the usefulness of these — 


heuristic concepts lies not in an extraphysical domain but 
rather depends on the degree to which they constitute a 
faithful code of experiential data. For example, a utility 
curve for money ts only useful so long as it reflects one’s 
experience regarding one’s ability to convert certain sums of 
money to streams of neural gratification. The rationale for 
using this utility curve to govern future actions is based on 
inductive projection of this past experience onto an estimate 
of the individual’s conversion abilities in the future (includ- 
ing sums of money not yet experienced), and should also be 
based on factual knowledge codified in probabilistic forms. 
Adherence to this paradigm of value should strip value 
judgments of their metaphysical, extrafactual status and 
reinstate them on equal footings with conditional prob- 
ability statements, vulnerable to empirical scrutiny and 
manageable by Bayesian technology. 


IL]. POTENTIAL APPLICATIONS 


In this section we shall explore some of the operational 
implications of the value paradigm developed in Section II. 
Two tasks are studied: validation and aggregation of value 
judgments. 


1 This idea, that human conception of value derives its structure from 
sensory information, goes back to John Locke. In his words, “‘And thus 
we see how Moral Being and Notions, are founded on, and terminated 
in these simple ideas, we have received from Sensation or Reflection, 
besides which, we have nothing at all in our Understanding, to employ 
our Thoughts about’’ [5, p. 195]. He further states, ““So that whenceso- 
ever, we take the Rule of Moral Actions; or what standard soever we 
frame in our Minds the ideas of Virtues or Vices, they consist only, and 
are made up of Collections of simple ideas which we originally 
received from Sense or Reflection’’ [5, p. 196]. 


Validation 

The problem of establishing an external means for 
demonstrating the effectiveness of decision-analytic methods 
is one of the most controversial issues in decision analysis 
[8]. The difficulty stems from the fact that by its very 
nature the idea of externally validating a value-processing 
procedure seems to imply a paradox: the proclaimed aim 
of such procedures 1s to remain as faithful as possible to the 
subjective preference structure of the value assessor. That 
structure, though, is only observable via the procedure 
under test, and so, value processing and decision-analytic 
procedures appears to be self-validated on a priori grounds. 

Our interpretation of value judgments, however, offers a 
sound resolution of this dilemma. No longer is fidelity with 
internal value structure an end in itself. One’s value struc- 
ture is merely a code of empirical data, and so, fidelity with 
that data should provide the necessary validation test. 

If one regards the individual value assessor as a trans- 
ducer for empirical data, then the objectives of value 
processing procedure are two-fold: first, gaining a faithful 
access to the code of experience and second, rectifying 
distortions between that code and the original data. Identi- 
cal objectives are assigned to probability-processing methods, 
coding distortions are handled by calibration, and elicita- 
tion distortions are minimized by the celebrated “divide 
and conquer” approach [9]. | 

Let us begin by examining this last approach more Salscy: 
It is devised to rectify a mismatch between the format in 
which a problem (or inquiry) is presented to an individual 
and the format in which that individual coded his experien- 
tial data. A model of the latter contains a combination of 
specific facts or scenarios (e.g., it rained on my last birthday), 
generic relations (e.g., when it rains I am normally moody) 
and procedures for assessing and integrating the former to 
provide answers to queries [10]. By and large, the formats. 
of the queries confronting an individual do not match the 
structure of his knowledge organization. For example, the 
query “Was I moody on my last birthday?” can be readily 
answered from the assumed stored knowledge. However, the 
query “Did it rain the last time I was moody?” requires 
further processing. This would be the case if my memory 
contains no explicit recollection of that last moody occasion, 
and if the memory file was not organized to be accessed by 
referring to my moods. An answer to such a query, though, 
can be reconstructed using knowledge fragments such as 
“When I have to write a paper I am almost always moody” | 
and ‘“‘Last season it hardly rained, and I had quite a few 
papers to write,” etc. The reconstruction procedure may be 
formalized in terms of the Bayes rule which provides the 
basis for the methodology of probabilistic information 
processing [11]. The rationale for this methodology lies 
with the assumption that judgments copied from memory 
are more reliable than those reconstructed from memory, 
and so, the methodology provides mechanical aids to 
perform the reconstruction on the basis of probabilistic 
judgments explicitly stored in one’s knowledge base. Note 
that not every division guarantees conquest, but only that 
which decomposes a query into a set of components which 
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are coded in explicit forms. Were J, for example, to decom- 
pose the query ““Was I moody last time it rained?” (assumed 
to be coded explicitly) into a set of queries concerning the 
weather characteristics conditioned upon my state of mood 
(not coded explicitly), the reliability of the reconstructed 
answer would only be diminished. 

Let us examine now if similar methodology could be 
applied to value elicitation. According to the paradigm 
_ proposed in Section ITI, value judgments require the mental 
calculation of E(S | 6), where @ is a description of a state of 
nature. It 1s clearly the case that no individual can expand 
the amount of memory necessary to store an explicit value 
of E(S|6@) for all conceivable @. Rather, people seem to 
configure their experience in a form of probabilistic rela- 
tions between S and various combinations of state attributes 
or aspects. For example, 0 may represent the possession of 
a specific house, while our experience with the benefits of 
house dwelling is coded in terms of the benefit derived from 
certain combinations of aspects such as privacy, locality, 
number of rooms, etc. These entities which serve as prim- 
itives for our value code are called value aspects, (the name 
“attributes” or “‘dimensions” are also used) and will be 
designated by the vector a = (a;,a>,°°°). In accordance 
with the “‘divide and conquer” principle, we should express 
the value U(@) in terms of a, which serves as a reference 
key to our code of knowledge. Thus we obtain 


U(6) = E(S| 0) = ¥ E(S| a,0)P(a | 8) (3) 


where P(a| @) represents the probability that aspect com- 
bination a would be attained assuming state 6 is reached. 
Since the quantities E(S | a@,@) are never found in one’s 
memory, we must replace them with E(S | a): 


U(8) = d E(S| a)P(a| 6) — 4) 


reflecting the fact that the value of any state @ can only be 
evaluated through its aspects, which in turn are defined by 
the language used in the organization of one’s knowledge 
base. 

If P(a| @) peaks sharply in the neighborhood of some 
aspect combination a(@), then (4) can be approximated by 


(5) 


For example, if a; represents the number of rooms in the 
example above, then P(a, | 0) is sharp since the number of 
rooms in a prospective house is usually known. Advanced 
knowledge of the degree of privacy (a,) provided by a cer- 
tain house, on the other hand, is generally less accurate and 
may require retaining the summation over P(a | 0) in (4). 
Among all techniques used in multiattribute utility 
analysis, the one most widely practiced and yet most hotly 
debated is the linear decomposition model [12], [13]. Here 
the analyst elicits the value (utility) of each attribute sep- 
arately and the relative importance of these attributes and 


: U(0) =~ E[S| a(0)]. 


then combines these in a linear fashion to arrive at an 


overall utility evaluation. In terms of our model, the issue 
of whether the linear model is adequate reduces to the 
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question of whether the conditional expectation E(S | «) 
can be decomposed in a linear fashion to give: 


E(S| a) = f,(a)) + fola) + °° © 


Formally speaking, such a decomposition can be guaranteed 
only when the a’s are both marginally and conditionally 
independent, a rare case for-an arbitrarily chosen set of 
attributes. One should recall though that a@ is not arbitrary 
but represents that selected collection of descriptors used 
by an individual to organize his experiential knowledge. 
The selection of these a’s has probably undergone a long pro- | 
cess of perceptual and linguistic evolution aimed at econo-— 
mizing computations. A linear decomposition as in (6) 
would save a substantial amount of storage space and time 
in the calculation of E(S | a). It stands to reason, therefore, 
that if a linear decomposition exists (using readily com- 
putable a’s), it would be chosen by the race and utilized in 
knowledge organization. 

All indications point to the fact that the attributes ase 
by people to represent knowledge have evolved in such a 
way as to render the decomposition in (6) feasible. Anderson 
[14] has found that in a wide variety of judgmental situa- 
tions (ranging from perceptual to linguistic experiments), 
human judgment of a unified ‘‘whole’”” can be modeled by 
very simple arithmetic combination (e.g., addition or 
multiplication) rules over the information components. 
Einhorn [15] and Dawes [16] reported that linear com- 
posities of attribute values elicited from individuals out- 
performed intuitive judgments obtained from the same 
individuals in such tasks as predicting a student’s college 
success or the longevity of patients with Hodgkins disease. 
Samuel [7] has written one of the most successful checker- 
playing programs which evaluates game positions by a 
linear combination of attributes (e.g., material advantage, 
mobility) defined by a human player. | 

These experiments seem to support the notion that most 
human knowledge is organized in terms of linearly decom- 
posable variables, Assuming now that value information is 
coded in a similar fashion, these experiments would con- 
stitute an external validation of the adequacy of the linear 
decomposition techniques. Indeed, it would be an extreme 
waste of computational resources for an organism to man- 
age two different forms of knowledge representation, one 
for factual knowledge, the other for normative knowledge. 
(Would knowledge about shoes be organized differently 
than knowledge about socks?) The superior performance 
of linear decomposition techniques in predicting factual 
data [13] provides an external validation for the effective- 
ness of multiattribute utility elicitation, although the latter 
is not observable directly. | 


Value Aggregation 


Bayesian techniques for aggregating probabilistic in- 


formation obtained from several assessors are based on the 


concept that such assessments represent outcomes of a 
noisy experiment which are probabilistically related to the 
underlying state of the world [3]. Let. r = (74,r2,°°*.r,) 
constitute a vector of probabilistic reports obtained from 
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experts regarding the probability of a specific variable 0. A 
straightforward application of the Bayes rule yields 


P(@|r) = kP(r| @)P(O). (7) 


The term P(r|@) formalizes the decisionmaker’s (or 
aggregator) knowledge regarding the manner in which the 
reports r are influenced by the underlying variable 6. It 
contains, for instance, his knowledge concerning the reputa- 
tion of various experts insofar as issuing a faithful and 
accurate report. It also contains knowledge about the 
correlation expected among the experts (e.g., it protects 
against counting repetitive reports with equal weights). 

The problem of integrating expert opinions is very similar 
to that of integrating knowledge received from various 
sections of one’s memory. The internal clues or attributes 
used to predict an outcome serve the same function as 
external experts; the two represent a condensed code of 
past experience which is no longer available to the inte- 
grator. There is one difference, though, which makes the 
problem of expert aggregation harder than that of internal 
aggregation. A person ts usually much more familiar with 
the nature of his internal procedures than with the nature 
of the experts he consults. For one thing, a person may have 
actively participated in the original design of his heuristic 
procedures (i.e., to render them linear decomposable) or 
may have observed their performance over a variety of 
inferential problems. In addition, tests on the operations 
of one’s internal procedures can be readily conducted, as 
they involve only mental exercises. As a result, the articula- 
tion of the function P(r| 0) may, in general, be a much 
harder task than the characterization of P(a | 0) or E(S | a) 
in (4). — 

Let us now imagine that a panel of experts provides a 
list of value judgments instead of a probabilistic report. 
For example, each of the ” experts may issue a real number 
V0), i = 1,°++,n, reflecting his assessment of the social 
value of a certain project 8. Our guiding paradigm dictates 
that each of these assessments should be treated as a con- 
densed code for the expert's personal experience with related 
events. It should be processed, therefore, as an outcome 
of a physical measurement on the underlying process 
_ relevant to the value of 0, in the same fashion as probabilistic 

reports. 3 

Let us first examine the case of competitive experts. As- 
_ sume that a leasing authority offers a certain parcel of un- 
- developed land for bid in the open market and that each 
bidder issues his personal judgment as to the value of the 
land. Such judgments are usually issued after a careful 
survey of the property and after assessing the manner in 
which the development project might fit into the bidder’s 
frame of business. It represents, therefore, the quantity 
V; which measures the expected benefit to the ith bidder, 
assuming he wins the contract. The leasing authority attempts 
to determine the real worth of the property V(@) on the 
bases of its own study as well as the aggregate reports it 
receives from the bidders. That worth depends on both the 
total income stream the property can generate, and the 
ability of the decisionmaker to enjoy that income. (Unlike 


a neural activity stream s(t), an income stream should be. 
discounted to account for uncertainties in one’s ability to 
consume the income [17] or convert it into neural satis- 
faction.) Assume, for the moment, that the decisionmaker’s 
expected benefit E(S'| ©) could be calculated given a set 
of attributes O = (0,,-°-,0;,) of the said property, (e.g., 
© could include the quantity of oil deposits, the depth of 
that oil, etc.). The value judgments V,,V.,---,V, could be 
used to modify the decisionmaker’s probability distribution 
on O, yielding an aggregate utility: 


U = EIS |) ViiVoee iV) 
ae 2, ECS ©)P(O | Vi5V2,° : “V) 


P(V,,V2,° ° “Vi | ©)P(O) 
. P(V,,°° Va) 


The term P(V,,---,V, | ©) represents, like the terms P(r | ©) 
in (7), the decisionmaker’s model of the experts, especially 
the dependence of their reports V,,°-:,V, on the value- 
determining parameter ©. It should reflect the professional 
reputation of their surveyors, a possible correlation among 
the latter, and even the possibility of a concerted attempt 
to underbid the agency. A Bayesian scheme identical to (8) 
was used by Kaplan [18] in analyzing off-shore oil bidding 
strategy. Its salient point is the exploitation of value judg-— 
ments as means for sharpening the decisionmaker’s estima- 
tion of a state parameter ©, which affects his own personal 
utility. 

Let us now return to a more cooperative situation and 
consider, for example, a public official polling his con- 
stituents regarding the value of a certain public policy. It is 
customary in problems of this sort to view the official as an 
echo to public wishes and to seek equitable entity called 
public utility, which should be obtained by some arithmetic 
aggregation rule applied to the individual values of the con- 
stituents [19]. Aside from the celebrated conceptual diffi-, 
culties connected with the concept of group utility [20], 
[21], this model also suffers from a practical drawback: 
it is utterly unrealistic. Regardless of how benevolent the 
public official is, he is bound by the very nature of his 
neural system wiring to act in a way that would maximize 
his own expected gratification. Part of this gratification is 
derived from materially contributing to public welfare, part 
from enjoying the image and reputation of a good-doer, and 
part from power-derived personal benefits. In short, it con- 
tains all factors and forces which play a role in the normal 
political arena. The maximization of the gratification func- 
tion S requires knowledge of certain parameters O which 
underly the sociopolitical scene. © could include, for ex- 
ample, the degree of public trust enjoyed by the official at 
any given time, his likelihood of retaining power, etc. A 
rational scheme of aggregating public values would there- 
fore consist of utilizing the latter as information signals to 
update one’s knowledge of © in a manner similar to that 
of (8). If, for example, a certain value-poll V,,---,V, 
strengthens the official’s hypothesis that the wishes of a 
certain vocal section of the public are erratic or transitory 
and therefore could be ignored, it makes both ethical and 
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logical sense for him to incorporate this knowledge into his 
action scheme. 

On this note we wish to argue that it is in the systemiza- 
tion of such knowledge-updating procedures, rather than 
the advocation of ad hoc value-aggregation rules that 
Bayes technology would provide a realistic assistance to 
the decision process. 


1V. SUMMARY 


The basic paradigm expanded in this paper is that value 
judgments are to be interpreted as representations of factual 
information. The meaning of such judgments, therefore, 
lies solely in their connection to empirical data recorded 
by an individual or by a cultured society. Viewed in this 
light, the problem of processing value judgments becomes 
almost identical to that of processing other codes of experi- 
ential knowledge, i.e., probabilistic judgments. This view 
implies that, aside from the desire to satify one’s value 
structure, other factors should also be considered, such as 
the faithfulness of the judgment and the appropriateness of 
the experience it represents to the facts surrounding the 
problems at hand. Bayes analysis, the formalism used for 
representing factual knowledge and probabilistic judgments, 
can also be used to capture such consideration of faithful- 
ness and, appropriateness of value judgments, and _ in- 
corporate the latter in a unified representation of human 
knowledge. 
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